[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:07 dist-test-slave-vd0p krb5kdc[15253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:07 dist-test-slave-vd0p krb5kdc[15253](info): set up 2 sockets
Sep 24 03:15:07 dist-test-slave-vd0p krb5kdc[15253](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 03:15:08.455421 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:08 dist-test-slave-vd0p krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:08.628070 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:39543
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:39543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:08.954048 15266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:08.954411 15266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:08.954598 15266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:08.961850 15266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:08.962040 15266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:08.962152 15266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:08.962284 15266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:08.962387 15266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:08.972211 15266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:39543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:08.973552 15266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:08.974938 15266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:08.987788 15271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:08.988443 15272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:08.988961 15274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:08.989770 15266 server_base.cc:1034] running on GCE node
I20260924 03:15:08.990643 15266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:08.992017 15266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:08.993248 15266 hybrid_clock.cc:648] HybridClock initialized: now 1790219708993183 us; error 32 us; skew 500 ppm
I20260924 03:15:08.993605 15266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:08 dist-test-slave-vd0p krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219708, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:08.996804 15266 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:08.998333 15266 webserver.cc:460] Webserver started at http://127.14.227.190:42657/ using document root <none> and password file <none>
I20260924 03:15:08.999023 15266 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:08.999164 15266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:08.999502 15266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:09.002964 15266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "ec89b0e4d85541619b8c6846256eef4b"
format_stamp: "Formatted at 2026-09-24 03:15:08 on dist-test-slave-vd0p"
server_key: "a8fa8f5e38a6b54b3cafc8a3711cf6af"
server_key_iv: "c99345a0214cb2e0b33dc418927d5a2d"
server_key_version: "encryptionkey@0"
I20260924 03:15:09.003717 15266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "ec89b0e4d85541619b8c6846256eef4b"
format_stamp: "Formatted at 2026-09-24 03:15:08 on dist-test-slave-vd0p"
server_key: "a8fa8f5e38a6b54b3cafc8a3711cf6af"
server_key_iv: "c99345a0214cb2e0b33dc418927d5a2d"
server_key_version: "encryptionkey@0"
I20260924 03:15:09.008756 15266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:15:09.012301 15281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:09.013597 15266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 03:15:09.013774 15266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "ec89b0e4d85541619b8c6846256eef4b"
format_stamp: "Formatted at 2026-09-24 03:15:08 on dist-test-slave-vd0p"
server_key: "a8fa8f5e38a6b54b3cafc8a3711cf6af"
server_key_iv: "c99345a0214cb2e0b33dc418927d5a2d"
server_key_version: "encryptionkey@0"
I20260924 03:15:09.014014 15266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:09.040340 15266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:09.044497 15266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:09.044777 15266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:09.063786 15266 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:39543
I20260924 03:15:09.063805 15332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:39543 every 8 connection(s)
I20260924 03:15:09.065100 15266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:09.068885 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:09.074074 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 15266
I20260924 03:15:09.074401 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:09.076792 15246 external_mini_cluster.cc:1428] Setting key 82d0a574128c9f611685e2895b36dc85
I20260924 03:15:09.080953 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b: Bootstrap starting.
I20260924 03:15:09.084092 15333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:09.085044 15333 log.cc:824] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:09.087809 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b: No bootstrap required, opened a new log
I20260924 03:15:09.097057 15333 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec89b0e4d85541619b8c6846256eef4b" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39543 } }
I20260924 03:15:09.097519 15333 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:09.097621 15333 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec89b0e4d85541619b8c6846256eef4b, State: Initialized, Role: FOLLOWER
I20260924 03:15:09.098273 15333 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec89b0e4d85541619b8c6846256eef4b" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39543 } }
I20260924 03:15:09.098511 15333 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:09.098675 15333 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:09.098829 15333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:09.101692 15333 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec89b0e4d85541619b8c6846256eef4b" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39543 } }
I20260924 03:15:09.102240 15333 leader_election.cc:302] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec89b0e4d85541619b8c6846256eef4b; no voters:
I20260924 03:15:09.102898 15333 leader_election.cc:288] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:09.103014 15338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:09.104511 15338 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [term 1 LEADER]: Becoming Leader. State: Replica: ec89b0e4d85541619b8c6846256eef4b, State: Running, Role: LEADER
I20260924 03:15:09.105039 15338 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec89b0e4d85541619b8c6846256eef4b" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39543 } }
I20260924 03:15:09.105262 15333 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 03:15:09 dist-test-slave-vd0p krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:09.107767 15339 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec89b0e4d85541619b8c6846256eef4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec89b0e4d85541619b8c6846256eef4b" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39543 } } }
I20260924 03:15:09.108127 15339 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:09.111438 15340 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec89b0e4d85541619b8c6846256eef4b. Latest consensus state: current_term: 1 leader_uuid: "ec89b0e4d85541619b8c6846256eef4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec89b0e4d85541619b8c6846256eef4b" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39543 } } }
I20260924 03:15:09.111850 15340 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:09.114055 15343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:09.119817 15343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:09.120514 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:09.081284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.14.227.190:39543)
0924 03:15:09.082272 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:09.082289 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:09.083568 (+ 1279us) server_negotiation.cc:407] Connection header received
0924 03:15:09.092549 (+ 8981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:09.092574 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:09.093064 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:09.093473 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:09.096160 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:09.098082 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:09.099918 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:09.100984 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:09.101162 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:09.108702 (+ 7540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:09.108717 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:09.108745 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:09.108814 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:09.114413 (+ 5599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:09.115549 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:09.115559 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:09.115570 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:09.115766 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:09.116219 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:09.116225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:09.116231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:09.117053 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:09.117397 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:09.118489 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0924 03:15:09.118969 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":465,"threads_started":1}
I20260924 03:15:09.133194 15343 catalog_manager.cc:1347] Generated new cluster ID: 1e1dde18cef34ab2aab4ee67e02b7fa5
I20260924 03:15:09.133378 15343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:09.147496 15343 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:09.149303 15343 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:09.165830 15343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec89b0e4d85541619b8c6846256eef4b: Generated new TSK 0
I20260924 03:15:09.166813 15343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:09.222049 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39543
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:09.558005 15359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:09.558275 15359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:09.558378 15359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:09.565500 15359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:09.565593 15359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:09.565762 15359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:09.575033 15359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:09.576205 15359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:09.577488 15359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:09.590405 15365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:09.590505 15364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:09.590488 15367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:09.591325 15359 server_base.cc:1034] running on GCE node
I20260924 03:15:09.591881 15359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:09.592751 15359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:09.594026 15359 hybrid_clock.cc:648] HybridClock initialized: now 1790219709593909 us; error 45 us; skew 500 ppm
I20260924 03:15:09.594343 15359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:09 dist-test-slave-vd0p krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219709, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:09.597414 15359 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:09.598896 15359 webserver.cc:460] Webserver started at http://127.14.227.129:32801/ using document root <none> and password file <none>
I20260924 03:15:09.599630 15359 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:09.599728 15359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:09.599982 15359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:09.602749 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "c0ad9baf9c8a4cdcbbb6329f4db8383c"
format_stamp: "Formatted at 2026-09-24 03:15:09 on dist-test-slave-vd0p"
server_key: "d36236c06dcf18245d4d2c300a3ac265"
server_key_iv: "5828748acf8f38e2fc8b17b60e2b6c90"
server_key_version: "encryptionkey@0"
I20260924 03:15:09.603454 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "c0ad9baf9c8a4cdcbbb6329f4db8383c"
format_stamp: "Formatted at 2026-09-24 03:15:09 on dist-test-slave-vd0p"
server_key: "d36236c06dcf18245d4d2c300a3ac265"
server_key_iv: "5828748acf8f38e2fc8b17b60e2b6c90"
server_key_version: "encryptionkey@0"
I20260924 03:15:09.608552 15359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:15:09.611955 15374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:09.613206 15359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:09.613380 15359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "c0ad9baf9c8a4cdcbbb6329f4db8383c"
format_stamp: "Formatted at 2026-09-24 03:15:09 on dist-test-slave-vd0p"
server_key: "d36236c06dcf18245d4d2c300a3ac265"
server_key_iv: "5828748acf8f38e2fc8b17b60e2b6c90"
server_key_version: "encryptionkey@0"
I20260924 03:15:09.613596 15359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:09.636445 15359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:09.640540 15359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:09.640782 15359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:09.641705 15359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:09.643044 15359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:09.643121 15359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:09.643203 15359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:09.643255 15359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:09.675884 15359 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:36571
I20260924 03:15:09.675958 15486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:36571 every 8 connection(s)
I20260924 03:15:09.677181 15359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:09.680573 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 15359
I20260924 03:15:09.680794 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:09.681430 15246 external_mini_cluster.cc:1428] Setting key f9481cea47e5320e7767061a2010e84f
Sep 24 03:15:09 dist-test-slave-vd0p krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219709, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:15:09.698450 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:09.679704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:39125 (local address 127.14.227.190:39543)
0924 03:15:09.680129 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:09.680133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:09.681207 (+ 1074us) server_negotiation.cc:407] Connection header received
0924 03:15:09.683459 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:09.683464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:09.683536 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:09.683645 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:09.685005 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:09.686227 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:09.688168 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:09.688543 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:09.688640 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:09.692542 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:09.692547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:09.692550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:09.692580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:09.695069 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:09.696008 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:09.696012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:09.696014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:09.696135 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:09.696617 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:09.696620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:09.696622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:09.696860 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:09.696972 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:09.697888 (+ 916us) server_negotiation.cc:299] Negotiation successful
0924 03:15:09.698152 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260924 03:15:09.699591 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:09.680267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:39543 (local address 127.14.227.129:39125)
0924 03:15:09.681040 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:09.681084 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:09.683177 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:09.683823 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:09.683831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:09.684296 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:09.684804 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:09.684819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:09.686373 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:09.686377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:09.688015 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:09.688031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:09.688727 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:09.688732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:09.688961 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:09.689643 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:09.689665 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:09.692318 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:09.695218 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:09.695230 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:09.695249 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:09.695888 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:09.696240 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:09.696245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:09.696250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:09.696507 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:09.696990 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:09.696998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:09.697477 (+ 479us) client_negotiation.cc:769] Sending connection context
0924 03:15:09.697822 (+ 345us) client_negotiation.cc:240] Negotiation successful
0924 03:15:09.698447 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":309,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:15:09.701156 15487 heartbeater.cc:347] Connected to a master server at 127.14.227.190:39543
I20260924 03:15:09.701474 15487 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:09.702225 15487 heartbeater.cc:511] Master 127.14.227.190:39543 requested a full tablet report, sending...
I20260924 03:15:09.704413 15298 ts_manager.cc:194] Registered new tserver with Master: c0ad9baf9c8a4cdcbbb6329f4db8383c (127.14.227.129:36571)
I20260924 03:15:09.706265 15298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:39125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:09.721463 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39543
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:10.035765 15493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:10.036039 15493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:10.036235 15493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:10.043416 15493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:10.043521 15493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:10.043701 15493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:10.053423 15493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:10.054759 15493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:10.056077 15493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:10.068884 15499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:10.068939 15498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:10.069803 15501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:10.070045 15493 server_base.cc:1034] running on GCE node
I20260924 03:15:10.070683 15493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:10.071642 15493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:10.072870 15493 hybrid_clock.cc:648] HybridClock initialized: now 1790219710072827 us; error 52 us; skew 500 ppm
I20260924 03:15:10.073189 15493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:10.076433 15493 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:10.078152 15493 webserver.cc:460] Webserver started at http://127.14.227.130:37847/ using document root <none> and password file <none>
I20260924 03:15:10.078807 15493 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:10.078956 15493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:10.079263 15493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:10.082154 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "ec3450ab3bec406c840f6156bccbd223"
format_stamp: "Formatted at 2026-09-24 03:15:10 on dist-test-slave-vd0p"
server_key: "cb138e455d27a6a47996af9e6febd4b7"
server_key_iv: "3a15b2a172158eaf448fea1f35972c57"
server_key_version: "encryptionkey@0"
I20260924 03:15:10.082886 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "ec3450ab3bec406c840f6156bccbd223"
format_stamp: "Formatted at 2026-09-24 03:15:10 on dist-test-slave-vd0p"
server_key: "cb138e455d27a6a47996af9e6febd4b7"
server_key_iv: "3a15b2a172158eaf448fea1f35972c57"
server_key_version: "encryptionkey@0"
I20260924 03:15:10.087852 15493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:15:10.091188 15508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:10.092447 15493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:15:10.092617 15493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "ec3450ab3bec406c840f6156bccbd223"
format_stamp: "Formatted at 2026-09-24 03:15:10 on dist-test-slave-vd0p"
server_key: "cb138e455d27a6a47996af9e6febd4b7"
server_key_iv: "3a15b2a172158eaf448fea1f35972c57"
server_key_version: "encryptionkey@0"
I20260924 03:15:10.092808 15493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:10.108807 15493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:10.112869 15493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:10.113171 15493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:10.114074 15493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:10.115361 15493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:10.115463 15493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:10.115609 15493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:10.115684 15493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:10.151831 15493 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:33221
I20260924 03:15:10.151875 15620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:33221 every 8 connection(s)
I20260924 03:15:10.153072 15493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:10.156421 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 15493
I20260924 03:15:10.156632 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:10.157171 15246 external_mini_cluster.cc:1428] Setting key e139a46f770d8c8e53bc85b445c1fe9d
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
I20260924 03:15:10.175540 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:10.155592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:50245 (local address 127.14.227.190:39543)
0924 03:15:10.155802 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:10.155808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:10.157182 (+ 1374us) server_negotiation.cc:407] Connection header received
0924 03:15:10.159530 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:10.159534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:10.159594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:10.159672 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:10.161787 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.162558 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.164521 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.165003 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.165108 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:10.168717 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:10.168726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:10.168734 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:10.168779 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:10.172579 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.173247 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.173254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.173258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.173418 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.173844 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.173850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.173857 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.174308 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:10.174471 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:10.175138 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 03:15:10.175309 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:15:10.176528 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:10.156138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:39543 (local address 127.14.227.130:50245)
0924 03:15:10.156997 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:10.157051 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:10.159240 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:10.159951 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:10.159964 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:10.160780 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:10.161598 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:10.161626 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.162724 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.162733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:10.164365 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:10.164377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.165187 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.165191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:10.165400 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:10.166107 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:10.166136 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:10.168480 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:10.172715 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:10.172722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:10.172735 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:10.173130 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:10.173535 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:10.173538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:10.173540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:10.173726 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:10.174431 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:10.174437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:10.174769 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 03:15:10.175033 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 03:15:10.175423 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":323,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
I20260924 03:15:10.178157 15621 heartbeater.cc:347] Connected to a master server at 127.14.227.190:39543
I20260924 03:15:10.178473 15621 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:10.179260 15621 heartbeater.cc:511] Master 127.14.227.190:39543 requested a full tablet report, sending...
I20260924 03:15:10.180981 15298 ts_manager.cc:194] Registered new tserver with Master: ec3450ab3bec406c840f6156bccbd223 (127.14.227.130:33221)
I20260924 03:15:10.181789 15298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:50245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:10.197284 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39543
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:10.522717 15626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:10.523012 15626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:10.523207 15626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:10.531291 15626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:10.531446 15626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:10.531652 15626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:10.541924 15626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:10.543241 15626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:10.544584 15626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:10.558012 15631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:10.557974 15632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:10.557992 15634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:10.559031 15626 server_base.cc:1034] running on GCE node
I20260924 03:15:10.559707 15626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:10.560720 15626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:10.561978 15626 hybrid_clock.cc:648] HybridClock initialized: now 1790219710561896 us; error 44 us; skew 500 ppm
I20260924 03:15:10.562312 15626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:10.565431 15626 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:10.567037 15626 webserver.cc:460] Webserver started at http://127.14.227.131:46061/ using document root <none> and password file <none>
I20260924 03:15:10.567739 15626 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:10.567883 15626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:10.568194 15626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:10.571178 15626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "4042a12625364585807d7c32082a57f6"
format_stamp: "Formatted at 2026-09-24 03:15:10 on dist-test-slave-vd0p"
server_key: "71862456adb1c8fcf9bbec337417a41d"
server_key_iv: "78a8a73aac7e179707ade9b7c152694d"
server_key_version: "encryptionkey@0"
I20260924 03:15:10.571933 15626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "4042a12625364585807d7c32082a57f6"
format_stamp: "Formatted at 2026-09-24 03:15:10 on dist-test-slave-vd0p"
server_key: "71862456adb1c8fcf9bbec337417a41d"
server_key_iv: "78a8a73aac7e179707ade9b7c152694d"
server_key_version: "encryptionkey@0"
I20260924 03:15:10.577085 15626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:15:10.580542 15641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:10.581848 15626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:10.582065 15626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "4042a12625364585807d7c32082a57f6"
format_stamp: "Formatted at 2026-09-24 03:15:10 on dist-test-slave-vd0p"
server_key: "71862456adb1c8fcf9bbec337417a41d"
server_key_iv: "78a8a73aac7e179707ade9b7c152694d"
server_key_version: "encryptionkey@0"
I20260924 03:15:10.582260 15626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:10.621315 15626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:10.625504 15626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:10.625751 15626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:10.626664 15626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:10.628041 15626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:10.628121 15626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:10.628204 15626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:10.628248 15626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:10.660719 15626 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:44229
I20260924 03:15:10.660778 15753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:44229 every 8 connection(s)
I20260924 03:15:10.662062 15626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:10.663283 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 15626
I20260924 03:15:10.663472 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:10.664032 15246 external_mini_cluster.cc:1428] Setting key 5bac0e7c879be2d6d391c6195e3d8e37
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:10.682253 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:10.664767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:58229 (local address 127.14.227.190:39543)
0924 03:15:10.664934 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:10.664939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:10.666475 (+ 1536us) server_negotiation.cc:407] Connection header received
0924 03:15:10.668520 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:10.668524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:10.668591 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:10.668670 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:10.670227 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.670963 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.672458 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.672850 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.672940 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:10.676267 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:10.676272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:10.676275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:10.676306 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:10.679264 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.680140 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.680146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.680150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.680310 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.680788 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.680792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.680794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.681054 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:10.681170 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:10.681778 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 03:15:10.681930 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:15:10.683146 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:10.665417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:39543 (local address 127.14.227.131:58229)
0924 03:15:10.666315 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:10.666366 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:10.668277 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:10.668875 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:10.668883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:10.669418 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:10.670068 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:10.670087 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.671084 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.671088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:10.672339 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:10.672350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.673021 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.673025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:10.673246 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:10.673861 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:10.673882 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:10.676062 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:10.679400 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:10.679409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:10.679425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:10.680003 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:10.680455 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:10.680460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:10.680463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:10.680672 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:10.681167 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:10.681173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:10.681465 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 03:15:10.681676 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 03:15:10.682105 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":427,"threads_started":1}
I20260924 03:15:10.684347 15754 heartbeater.cc:347] Connected to a master server at 127.14.227.190:39543
I20260924 03:15:10.684633 15754 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:10.685312 15754 heartbeater.cc:511] Master 127.14.227.190:39543 requested a full tablet report, sending...
I20260924 03:15:10.686686 15298 ts_manager.cc:194] Registered new tserver with Master: 4042a12625364585807d7c32082a57f6 (127.14.227.131:44229)
I20260924 03:15:10.687492 15298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:58229
I20260924 03:15:10.692415 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:10.709108 15487 heartbeater.cc:503] Master 127.14.227.190:39543 was elected leader, sending a full tablet report...
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:10.735716 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:10.721145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.14.227.129:36571)
0924 03:15:10.721765 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:10.721772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:10.721800 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:15:10.721884 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:10.721890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:10.722168 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:10.722305 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:10.723612 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.724651 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.725826 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.726436 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.726531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:10.730320 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:10.730331 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:10.730347 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:10.730382 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:10.733206 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.733783 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.733790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.733793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.733918 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.734374 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.734378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.734380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.734727 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:10.734948 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:10.735209 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:15:10.735411 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":312,"threads_started":1}
W20260924 03:15:10.737013 15402 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43020
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:10.778733 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:10.766099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.14.227.129:36571)
0924 03:15:10.766342 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:10.766349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:10.766501 (+ 152us) server_negotiation.cc:407] Connection header received
0924 03:15:10.766648 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:10.766652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:10.766741 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:10.766828 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:10.768079 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.768735 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.769868 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.770335 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.770424 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:10.774006 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:10.774011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:10.774015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:10.774043 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:10.776513 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.777110 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.777113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.777115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.777233 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.777522 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.777524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.777526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.777752 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:10.777922 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:10.778231 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 03:15:10.778383 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260924 03:15:10.780239 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 15359
I20260924 03:15:10.792174 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:10.781196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.14.227.129:36571)
0924 03:15:10.781397 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:10.781403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:10.781636 (+ 233us) server_negotiation.cc:407] Connection header received
0924 03:15:10.781732 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:10.781736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:10.781786 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:10.781857 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:10.782963 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.783610 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.784945 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.785386 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.785474 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:10.787637 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:10.787641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:10.787645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:10.787671 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:10.790131 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.790676 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.790679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.790681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.790816 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.791093 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.791096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.791098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.791321 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:10.791483 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:10.791680 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 03:15:10.791818 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:15:10.792851 15482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:10.875845 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 15359
I20260924 03:15:10.891228 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 15493
Sep 24 03:15:10 dist-test-slave-vd0p krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:15:10.906628 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:10.892020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.14.227.130:33221)
0924 03:15:10.892617 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:10.892624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:10.892651 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:15:10.892713 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:10.892719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:10.892933 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:10.893077 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:10.894263 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.895268 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.896500 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:10.897162 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:10.897262 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:10.901265 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:10.901276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:10.901291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:10.901323 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:10.904007 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.904578 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.904584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.904586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.904708 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:10.905045 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:10.905048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:10.905050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:10.905381 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:10.905725 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:10.905851 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 03:15:10.906301 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":315,"threads_started":1}
I20260924 03:15:10.907856 15616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:10.988987 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 15493
I20260924 03:15:11.004249 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 15626
Sep 24 03:15:11 dist-test-slave-vd0p krb5kdc[15253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:15:11.020749 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:11.005291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.14.227.131:44229)
0924 03:15:11.006038 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:11.006047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:11.006086 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:15:11.006146 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:11.006152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:11.006368 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:11.006541 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:11.007763 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:11.008783 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:11.010224 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:11.010862 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:11.011011 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:11.015069 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:11.015080 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:11.015094 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:11.015129 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:11.017911 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:11.018648 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:11.018659 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:11.018662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:11.018803 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:11.019179 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:11.019186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:11.019190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:11.019517 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:11.019846 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:11.020133 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 03:15:11.020405 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":350,"threads_started":1}
I20260924 03:15:11.022032 15749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:11.102252 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 15626
I20260924 03:15:11.117859 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 15266
I20260924 03:15:11.119036 15328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:11.220585 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 15266
2026-09-24T03:15:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3859 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:11 dist-test-slave-vd0p krb5kdc[15775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:11 dist-test-slave-vd0p krb5kdc[15775](info): set up 2 sockets
Sep 24 03:15:11 dist-test-slave-vd0p krb5kdc[15775](info): commencing operation
krb5kdc: starting...
W20260924 03:15:12.377926 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:12 dist-test-slave-vd0p krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:12.545152 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:39079
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:12.849852 15788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:12.850160 15788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:12.850257 15788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:12.857276 15788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:12.857378 15788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:12.857424 15788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:12.857479 15788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:12.857522 15788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:12.866997 15788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:39079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:12.868175 15788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:12.869341 15788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:12.880512 15794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:12.880599 15796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:12.880577 15793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:12.881377 15788 server_base.cc:1034] running on GCE node
I20260924 03:15:12.882172 15788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:12.883503 15788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:12.884732 15788 hybrid_clock.cc:648] HybridClock initialized: now 1790219712884691 us; error 50 us; skew 500 ppm
I20260924 03:15:12.885064 15788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:12 dist-test-slave-vd0p krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219712, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:12.892702 15788 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:12.894273 15788 webserver.cc:460] Webserver started at http://127.14.227.190:41203/ using document root <none> and password file <none>
I20260924 03:15:12.894891 15788 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:12.894997 15788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:12.895258 15788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:12.898609 15788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "4dc6cc813ea344eaa105838703c20f7f"
format_stamp: "Formatted at 2026-09-24 03:15:12 on dist-test-slave-vd0p"
server_key: "39606e8d23dc3c589f25e45b39e4ad6e"
server_key_iv: "983240843da57320d83abf19bd2f91e9"
server_key_version: "encryptionkey@0"
I20260924 03:15:12.899314 15788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "4dc6cc813ea344eaa105838703c20f7f"
format_stamp: "Formatted at 2026-09-24 03:15:12 on dist-test-slave-vd0p"
server_key: "39606e8d23dc3c589f25e45b39e4ad6e"
server_key_iv: "983240843da57320d83abf19bd2f91e9"
server_key_version: "encryptionkey@0"
I20260924 03:15:12.904347 15788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:15:12.907743 15803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:12.909065 15788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:12.909243 15788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "4dc6cc813ea344eaa105838703c20f7f"
format_stamp: "Formatted at 2026-09-24 03:15:12 on dist-test-slave-vd0p"
server_key: "39606e8d23dc3c589f25e45b39e4ad6e"
server_key_iv: "983240843da57320d83abf19bd2f91e9"
server_key_version: "encryptionkey@0"
I20260924 03:15:12.909446 15788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:12.937703 15788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:12.942137 15788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:12.942384 15788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:12.960258 15788 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:39079
I20260924 03:15:12.960266 15854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:39079 every 8 connection(s)
I20260924 03:15:12.961627 15788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:12.965157 15855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:12.968850 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 15788
I20260924 03:15:12.969077 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:12.969903 15246 external_mini_cluster.cc:1428] Setting key 134a44a709f61672b50fce7113ce8744
I20260924 03:15:12.978713 15855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f: Bootstrap starting.
I20260924 03:15:12.982483 15855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:12.983831 15855 log.cc:824] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:15:12 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:12.987068 15855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f: No bootstrap required, opened a new log
I20260924 03:15:12.993069 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:12.972047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.14.227.190:39079)
0924 03:15:12.972813 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:12.972824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:12.972882 (+ 58us) server_negotiation.cc:407] Connection header received
0924 03:15:12.974561 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:12.974584 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:12.974998 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:12.975350 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:12.977122 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:12.978480 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:12.980152 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:12.980832 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:12.980970 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:12.985828 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:12.985835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:12.985849 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:12.985881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:12.989040 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:12.989823 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:12.989828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:12.989834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:12.990060 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:12.990430 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:12.990433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:12.990435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:12.990942 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:12.991191 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:12.991700 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 03:15:12.992065 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":385,"threads_started":1}
I20260924 03:15:12.996380 15855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc6cc813ea344eaa105838703c20f7f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39079 } }
I20260924 03:15:12.996731 15855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:12.996850 15855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc6cc813ea344eaa105838703c20f7f, State: Initialized, Role: FOLLOWER
I20260924 03:15:12.997431 15855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc6cc813ea344eaa105838703c20f7f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39079 } }
I20260924 03:15:12.997674 15855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:12.997805 15855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:12.998000 15855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:13.000304 15855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc6cc813ea344eaa105838703c20f7f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39079 } }
I20260924 03:15:13.000782 15855 leader_election.cc:302] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dc6cc813ea344eaa105838703c20f7f; no voters:
I20260924 03:15:13.001338 15855 leader_election.cc:288] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:13.001473 15860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:13.002162 15860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [term 1 LEADER]: Becoming Leader. State: Replica: 4dc6cc813ea344eaa105838703c20f7f, State: Running, Role: LEADER
I20260924 03:15:13.002699 15860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc6cc813ea344eaa105838703c20f7f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39079 } }
I20260924 03:15:13.003367 15855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:13.005892 15862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dc6cc813ea344eaa105838703c20f7f. Latest consensus state: current_term: 1 leader_uuid: "4dc6cc813ea344eaa105838703c20f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc6cc813ea344eaa105838703c20f7f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39079 } } }
I20260924 03:15:13.006428 15862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:13.006317 15861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dc6cc813ea344eaa105838703c20f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc6cc813ea344eaa105838703c20f7f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 39079 } } }
I20260924 03:15:13.007050 15861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:13.012243 15869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:13.017267 15869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:13.025141 15869 catalog_manager.cc:1347] Generated new cluster ID: 544173e1acbb40de844c124b364a50f8
I20260924 03:15:13.025267 15869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:13.061858 15869 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:13.063161 15869 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:13.068956 15869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dc6cc813ea344eaa105838703c20f7f: Generated new TSK 0
I20260924 03:15:13.069751 15869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:13.109333 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39079
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:13.428252 15881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:13.428522 15881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:13.428694 15881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:13.436008 15881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:13.436103 15881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:13.436295 15881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:13.446020 15881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:13.447266 15881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:13.448556 15881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:13.461231 15889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:13.461340 15886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:13.461297 15887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:13.462301 15881 server_base.cc:1034] running on GCE node
I20260924 03:15:13.462925 15881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:13.463855 15881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:13.465040 15881 hybrid_clock.cc:648] HybridClock initialized: now 1790219713464985 us; error 50 us; skew 500 ppm
I20260924 03:15:13.465358 15881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:13 dist-test-slave-vd0p krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219713, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:13.469466 15881 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:13.471640 15881 webserver.cc:460] Webserver started at http://127.14.227.129:33761/ using document root <none> and password file <none>
I20260924 03:15:13.472538 15881 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:13.472689 15881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:13.473003 15881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:13.476480 15881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "608b3f57a29748888424b20fc28ae2d3"
format_stamp: "Formatted at 2026-09-24 03:15:13 on dist-test-slave-vd0p"
server_key: "4a56cdd826bb4688f3feb08fbd0b5d95"
server_key_iv: "e555903bfb1b33c2559d0d02f8e68220"
server_key_version: "encryptionkey@0"
I20260924 03:15:13.477200 15881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "608b3f57a29748888424b20fc28ae2d3"
format_stamp: "Formatted at 2026-09-24 03:15:13 on dist-test-slave-vd0p"
server_key: "4a56cdd826bb4688f3feb08fbd0b5d95"
server_key_iv: "e555903bfb1b33c2559d0d02f8e68220"
server_key_version: "encryptionkey@0"
I20260924 03:15:13.482198 15881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 03:15:13.485582 15896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:13.486965 15881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:13.487102 15881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "608b3f57a29748888424b20fc28ae2d3"
format_stamp: "Formatted at 2026-09-24 03:15:13 on dist-test-slave-vd0p"
server_key: "4a56cdd826bb4688f3feb08fbd0b5d95"
server_key_iv: "e555903bfb1b33c2559d0d02f8e68220"
server_key_version: "encryptionkey@0"
I20260924 03:15:13.487277 15881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:13.497735 15881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:13.501484 15881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:13.501730 15881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:13.502710 15881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:13.504070 15881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:13.504153 15881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:13.504256 15881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:13.504328 15881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:13.535630 15881 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:33229
I20260924 03:15:13.535676 16008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:33229 every 8 connection(s)
I20260924 03:15:13.536890 15881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:13.545233 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 15881
I20260924 03:15:13.545441 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:13.546082 15246 external_mini_cluster.cc:1428] Setting key 607ce7f20c916ca2d9d49aa5972177bf
Sep 24 03:15:13 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219713, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:13.557894 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:13.539400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:57297 (local address 127.14.227.190:39079)
0924 03:15:13.539827 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:13.539832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:13.540745 (+ 913us) server_negotiation.cc:407] Connection header received
0924 03:15:13.542591 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:13.542596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:13.542679 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:13.542811 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:13.544226 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:13.545005 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:13.546783 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:13.547547 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:13.547653 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:13.551632 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:13.551659 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:13.551668 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:13.551716 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:13.554611 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:13.555527 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:13.555531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:13.555533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:13.555644 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:13.556155 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:13.556159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:13.556161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:13.556448 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:13.556565 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:13.557447 (+ 882us) server_negotiation.cc:299] Negotiation successful
0924 03:15:13.557609 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":200,"threads_started":1}
I20260924 03:15:13.559533 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:13.539944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:39079 (local address 127.14.227.129:57297)
0924 03:15:13.540597 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:13.540633 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:13.542339 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:13.543052 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:13.543060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:13.543542 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:13.544064 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:13.544079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:13.545150 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:13.545160 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:13.546601 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:13.546616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:13.547782 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:13.547789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:13.548120 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:13.548970 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:13.548991 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:13.551376 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:13.554745 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:13.554758 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:13.554778 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:13.555406 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:13.555742 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:13.555750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:13.555757 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:13.556001 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:13.556584 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:13.556595 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:13.557051 (+ 456us) client_negotiation.cc:769] Sending connection context
0924 03:15:13.557367 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 03:15:13.557986 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":251,"threads_started":1}
I20260924 03:15:13.561152 16009 heartbeater.cc:347] Connected to a master server at 127.14.227.190:39079
I20260924 03:15:13.561510 16009 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:13.562242 16009 heartbeater.cc:511] Master 127.14.227.190:39079 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:15:13.564601 15820 ts_manager.cc:194] Registered new tserver with Master: 608b3f57a29748888424b20fc28ae2d3 (127.14.227.129:33229)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:15:13.566398 15820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:57297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:13.586452 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39079
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:13.901463 16015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:13.901774 16015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:13.901898 16015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:13.909209 16015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:13.909322 16015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:13.909518 16015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:13.919304 16015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:13.920550 16015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:13.921995 16015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:13.935300 16021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:13.935298 16023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:13.935432 16020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:13.935887 16015 server_base.cc:1034] running on GCE node
I20260924 03:15:13.936508 16015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:13.937430 16015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:13.938676 16015 hybrid_clock.cc:648] HybridClock initialized: now 1790219713938614 us; error 48 us; skew 500 ppm
I20260924 03:15:13.939023 16015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:13 dist-test-slave-vd0p krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219713, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:13.942091 16015 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:13.943465 16015 webserver.cc:460] Webserver started at http://127.14.227.130:45303/ using document root <none> and password file <none>
I20260924 03:15:13.944152 16015 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:13.944294 16015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:13.944599 16015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:13.947495 16015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "84d407aa1b454c9b9b6a7dce2df31e4b"
format_stamp: "Formatted at 2026-09-24 03:15:13 on dist-test-slave-vd0p"
server_key: "f12654c57614666d06a49e83745156a0"
server_key_iv: "b0e4e7bc54d3b1db7e2b7812a1f4b861"
server_key_version: "encryptionkey@0"
I20260924 03:15:13.948263 16015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "84d407aa1b454c9b9b6a7dce2df31e4b"
format_stamp: "Formatted at 2026-09-24 03:15:13 on dist-test-slave-vd0p"
server_key: "f12654c57614666d06a49e83745156a0"
server_key_iv: "b0e4e7bc54d3b1db7e2b7812a1f4b861"
server_key_version: "encryptionkey@0"
I20260924 03:15:13.953253 16015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 03:15:13.956677 16030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:13.958022 16015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:13.958181 16015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "84d407aa1b454c9b9b6a7dce2df31e4b"
format_stamp: "Formatted at 2026-09-24 03:15:13 on dist-test-slave-vd0p"
server_key: "f12654c57614666d06a49e83745156a0"
server_key_iv: "b0e4e7bc54d3b1db7e2b7812a1f4b861"
server_key_version: "encryptionkey@0"
I20260924 03:15:13.958374 16015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:13.977020 16015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:13.980923 16015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:13.981208 16015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:13.982252 16015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:13.983604 16015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:13.983704 16015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:13.983840 16015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:13.983915 16015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:14.016010 16015 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:38285
I20260924 03:15:14.016052 16142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:38285 every 8 connection(s)
I20260924 03:15:14.017307 16015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:14.022440 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 16015
I20260924 03:15:14.022635 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:14.023061 15246 external_mini_cluster.cc:1428] Setting key db0c7eef5c3e4c472c8eb4a95e7b7c8a
Sep 24 03:15:14 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219713, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:14.039559 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.019927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:35967 (local address 127.14.227.190:39079)
0924 03:15:14.020165 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.020169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.021285 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 03:15:14.023272 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.023278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.023364 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.023486 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:14.025389 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.026509 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.028026 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.028760 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.028912 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.032212 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:14.032221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:14.032229 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:14.032275 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:14.036061 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.037009 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.037014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.037016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.037144 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.037708 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.037712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.037715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.038107 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:14.038252 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.039159 (+ 907us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.039327 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
I20260924 03:15:14.041074 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.020420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:39079 (local address 127.14.227.130:35967)
0924 03:15:14.021118 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:14.021157 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:14.022986 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:14.023699 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:14.023710 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:14.024408 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:14.025229 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.025252 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.026649 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.026653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.027886 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.027898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.028958 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.028963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.029155 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.029852 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:14.029878 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:14.032006 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:14.036190 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:14.036196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:14.036209 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:14.036872 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:14.037261 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:14.037269 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:14.037274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:14.037536 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:14.038260 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:14.038270 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:14.038719 (+ 449us) client_negotiation.cc:769] Sending connection context
0924 03:15:14.039086 (+ 367us) client_negotiation.cc:240] Negotiation successful
0924 03:15:14.039681 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":282,"threads_started":1}
I20260924 03:15:14.042523 16143 heartbeater.cc:347] Connected to a master server at 127.14.227.190:39079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
I20260924 03:15:14.042868 16143 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:14.043552 16143 heartbeater.cc:511] Master 127.14.227.190:39079 requested a full tablet report, sending...
I20260924 03:15:14.044961 15820 ts_manager.cc:194] Registered new tserver with Master: 84d407aa1b454c9b9b6a7dce2df31e4b (127.14.227.130:38285)
I20260924 03:15:14.045780 15820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:35967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:14.064584 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39079
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:14.376544 16148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:14.376816 16148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:14.376974 16148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:14.383997 16148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:14.384104 16148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:14.384281 16148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:14.394052 16148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:39079
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:14.395323 16148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:14.396636 16148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:14.409222 16153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:14.409417 16156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:14.409252 16154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:14.410097 16148 server_base.cc:1034] running on GCE node
I20260924 03:15:14.410730 16148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:14.411544 16148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:14.412787 16148 hybrid_clock.cc:648] HybridClock initialized: now 1790219714412735 us; error 39 us; skew 500 ppm
I20260924 03:15:14.413079 16148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:14 dist-test-slave-vd0p krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219714, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:14.416165 16148 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:14.417673 16148 webserver.cc:460] Webserver started at http://127.14.227.131:35005/ using document root <none> and password file <none>
I20260924 03:15:14.418383 16148 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:14.418519 16148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:14.418821 16148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:14.421512 16148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "8ddc30ad902e462aac94446c564aed28"
format_stamp: "Formatted at 2026-09-24 03:15:14 on dist-test-slave-vd0p"
server_key: "aef884a3d833ada2d83d90357be573ef"
server_key_iv: "3a8df511173f16d294620369c596395f"
server_key_version: "encryptionkey@0"
I20260924 03:15:14.422333 16148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "8ddc30ad902e462aac94446c564aed28"
format_stamp: "Formatted at 2026-09-24 03:15:14 on dist-test-slave-vd0p"
server_key: "aef884a3d833ada2d83d90357be573ef"
server_key_iv: "3a8df511173f16d294620369c596395f"
server_key_version: "encryptionkey@0"
I20260924 03:15:14.427104 16148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:15:14.430438 16163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:14.431697 16148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:15:14.431859 16148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "8ddc30ad902e462aac94446c564aed28"
format_stamp: "Formatted at 2026-09-24 03:15:14 on dist-test-slave-vd0p"
server_key: "aef884a3d833ada2d83d90357be573ef"
server_key_iv: "3a8df511173f16d294620369c596395f"
server_key_version: "encryptionkey@0"
I20260924 03:15:14.432042 16148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:14.454375 16148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:14.458487 16148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:14.458762 16148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:14.459702 16148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:14.460959 16148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:14.461063 16148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:14.461199 16148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:14.461266 16148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:14.494505 16148 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:45949
I20260924 03:15:14.494556 16275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:45949 every 8 connection(s)
I20260924 03:15:14.495820 16148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:14.499365 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 16148
I20260924 03:15:14.499555 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:14.500094 15246 external_mini_cluster.cc:1428] Setting key 84d2ae89f2198788f217ba1f51cf59c5
Sep 24 03:15:14 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219714, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:14.516022 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.498321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:48015 (local address 127.14.227.190:39079)
0924 03:15:14.498489 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.498493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.499796 (+ 1303us) server_negotiation.cc:407] Connection header received
0924 03:15:14.501898 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.501903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.502021 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.502112 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:14.504214 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.505000 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.506580 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.507009 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.507102 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.510315 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:14.510320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:14.510323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:14.510352 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:14.512968 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.513684 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.513688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.513691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.513824 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.514242 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.514245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.514247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.514498 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:14.514647 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.515520 (+ 873us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.515681 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:15:14.517315 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.498809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:39079 (local address 127.14.227.131:48015)
0924 03:15:14.499610 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:14.499675 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:14.501563 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:14.502400 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:14.502414 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:14.503221 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:14.504021 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.504046 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.505137 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.505140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.506448 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.506459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.507177 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.507181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.507372 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.508008 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:14.508029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:14.510092 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:14.513083 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:14.513090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:14.513120 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:14.513583 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:14.513910 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:14.513913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:14.513916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:14.514135 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:14.514648 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:14.514658 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:14.515063 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 03:15:14.515408 (+ 345us) client_negotiation.cc:240] Negotiation successful
0924 03:15:14.515974 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":293,"threads_started":1}
I20260924 03:15:14.518534 16276 heartbeater.cc:347] Connected to a master server at 127.14.227.190:39079
I20260924 03:15:14.518823 16276 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:14.519497 16276 heartbeater.cc:511] Master 127.14.227.190:39079 requested a full tablet report, sending...
I20260924 03:15:14.520830 15820 ts_manager.cc:194] Registered new tserver with Master: 8ddc30ad902e462aac94446c564aed28 (127.14.227.131:45949)
I20260924 03:15:14.521596 15820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:48015
I20260924 03:15:14.528503 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:14.548974 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.534592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.14.227.190:39079)
0924 03:15:14.534795 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.534803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.535507 (+ 704us) server_negotiation.cc:407] Connection header received
0924 03:15:14.535749 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.535755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.535846 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.535960 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:14.538138 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.538882 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.541012 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.541426 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.541518 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.544172 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:14.544176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:14.544179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:14.544209 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:14.546787 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.547369 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.547372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.547374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.547489 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.547779 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.547782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.547784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.548056 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:14.548226 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.548492 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.548641 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:15:14.554934 15820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:14.561607 15820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:14.569916 16009 heartbeater.cc:503] Master 127.14.227.190:39079 was elected leader, sending a full tablet report...
I20260924 03:15:14.589545 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.573397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:33229 (local address 127.0.0.1:39134)
0924 03:15:14.574477 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:14.574521 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:14.574713 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:14.581069 (+ 6356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:14.581077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:14.581116 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:14.581503 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.581523 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.583754 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.583761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.586460 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.586478 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.588509 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.588515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.588779 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.588829 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 03:15:14.588965 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 03:15:14.589141 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":1,"thread_start_us":6567,"threads_started":1}
I20260924 03:15:14.589557 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.573639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.14.227.129:33229)
0924 03:15:14.580546 (+ 6907us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.580555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.580584 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:15:14.580678 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.580684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.580869 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.581066 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:14.581762 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.583577 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.586690 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.588290 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.588476 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.588692 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.589010 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.589199 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6692,"thread_start_us":6521,"threads_started":1}
I20260924 03:15:14.591806 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.573739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:45949 (local address 127.0.0.1:59492)
0924 03:15:14.580290 (+ 6551us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:14.580311 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:14.580470 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:14.581365 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:14.581376 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:14.581403 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:14.582996 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.583011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.585201 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.585207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.588737 (+ 3530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.588752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.590613 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.590620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.590798 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.590848 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 03:15:14.591148 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 03:15:14.591295 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6402,"mutex_wait_us":32,"thread_start_us":979,"threads_started":1}
I20260924 03:15:14.592574 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.574382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.14.227.131:45949)
0924 03:15:14.579940 (+ 5558us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.579953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.580438 (+ 485us) server_negotiation.cc:407] Connection header received
0924 03:15:14.580656 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.580664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.580945 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.581168 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:14.583240 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.585007 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.588903 (+ 3896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.590383 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.590616 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.590799 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.592063 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.592270 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5353,"thread_start_us":5197,"threads_started":1}
I20260924 03:15:14.593974 15944 tablet_service.cc:1467] Processing CreateTablet for tablet acd6eda54d134a2db7a8fd075a48868b (DEFAULT_TABLE table=test-table [id=0ab67c5070194086a0766d75522236c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:14.595214 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.581644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:38285 (local address 127.0.0.1:59472)
0924 03:15:14.587270 (+ 5626us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:14.587284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:14.587428 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:14.587960 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:14.587965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:14.587990 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:14.588397 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.588410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.590365 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.590369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.592451 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.592465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.594299 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.594306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.594480 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.594517 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:15:14.594637 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 03:15:14.594770 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5508,"mutex_wait_us":1,"thread_start_us":5349,"threads_started":1}
I20260924 03:15:14.595228 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.580641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.14.227.130:38285)
0924 03:15:14.586907 (+ 6266us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.586919 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.587385 (+ 466us) server_negotiation.cc:407] Connection header received
0924 03:15:14.587588 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.587594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.587786 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.587979 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:14.588606 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.590180 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.592629 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.594097 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.594284 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.594461 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.594699 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.594864 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6110,"thread_start_us":6013,"threads_started":1}
I20260924 03:15:14.595851 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd6eda54d134a2db7a8fd075a48868b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:14.596210 16211 tablet_service.cc:1467] Processing CreateTablet for tablet acd6eda54d134a2db7a8fd075a48868b (DEFAULT_TABLE table=test-table [id=0ab67c5070194086a0766d75522236c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:14.597590 16211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd6eda54d134a2db7a8fd075a48868b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:14.600260 16078 tablet_service.cc:1467] Processing CreateTablet for tablet acd6eda54d134a2db7a8fd075a48868b (DEFAULT_TABLE table=test-table [id=0ab67c5070194086a0766d75522236c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:14.601294 16078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd6eda54d134a2db7a8fd075a48868b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:14.610637 16294 tablet_bootstrap.cc:492] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3: Bootstrap starting.
I20260924 03:15:14.615239 16296 tablet_bootstrap.cc:492] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28: Bootstrap starting.
I20260924 03:15:14.616366 16294 tablet_bootstrap.cc:654] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:14.617916 16296 tablet_bootstrap.cc:654] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:14.618081 16294 log.cc:824] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:14.619194 16296 log.cc:824] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:14.619318 16295 tablet_bootstrap.cc:492] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b: Bootstrap starting.
I20260924 03:15:14.623068 16295 tablet_bootstrap.cc:654] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:14.624290 16295 log.cc:824] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:14.624521 16296 tablet_bootstrap.cc:492] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28: No bootstrap required, opened a new log
I20260924 03:15:14.624634 16294 tablet_bootstrap.cc:492] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3: No bootstrap required, opened a new log
I20260924 03:15:14.624805 16296 ts_tablet_manager.cc:1397] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 03:15:14.624979 16294 ts_tablet_manager.cc:1397] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260924 03:15:14.627771 16295 tablet_bootstrap.cc:492] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b: No bootstrap required, opened a new log
I20260924 03:15:14.628005 16295 ts_tablet_manager.cc:1397] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 03:15:14.632172 16296 raft_consensus.cc:348] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.632516 16296 raft_consensus.cc:374] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:14.632616 16296 raft_consensus.cc:729] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddc30ad902e462aac94446c564aed28, State: Initialized, Role: FOLLOWER
I20260924 03:15:14.633315 16296 consensus_queue.cc:260] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.634356 16276 heartbeater.cc:503] Master 127.14.227.190:39079 was elected leader, sending a full tablet report...
I20260924 03:15:14.635390 16296 ts_tablet_manager.cc:1428] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 03:15:14.635408 16294 raft_consensus.cc:348] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.635867 16294 raft_consensus.cc:374] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:14.636009 16294 raft_consensus.cc:729] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608b3f57a29748888424b20fc28ae2d3, State: Initialized, Role: FOLLOWER
I20260924 03:15:14.636611 16294 consensus_queue.cc:260] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.638715 16295 raft_consensus.cc:348] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.639281 16295 raft_consensus.cc:374] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:14.639402 16295 raft_consensus.cc:729] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d407aa1b454c9b9b6a7dce2df31e4b, State: Initialized, Role: FOLLOWER
I20260924 03:15:14.640048 16295 consensus_queue.cc:260] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.641135 16143 heartbeater.cc:503] Master 127.14.227.190:39079 was elected leader, sending a full tablet report...
I20260924 03:15:14.642069 16295 ts_tablet_manager.cc:1428] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b: Time spent starting tablet: real 0.014s user 0.009s sys 0.007s
I20260924 03:15:14.644505 16294 ts_tablet_manager.cc:1428] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3: Time spent starting tablet: real 0.019s user 0.007s sys 0.010s
W20260924 03:15:14.747378 16277 tablet.cc:2369] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:15:14.769476 16144 tablet.cc:2369] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:15:14.789337 16010 tablet.cc:2369] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:14.798344 16301 raft_consensus.cc:482] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:14.798594 16301 raft_consensus.cc:504] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.800139 16301 leader_election.cc:288] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ddc30ad902e462aac94446c564aed28 (127.14.227.131:45949), 84d407aa1b454c9b9b6a7dce2df31e4b (127.14.227.130:38285)
I20260924 03:15:14.809147 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.800643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:52319 (local address 127.14.227.131:45949)
0924 03:15:14.800901 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.800910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.801352 (+ 442us) server_negotiation.cc:407] Connection header received
0924 03:15:14.801592 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.801598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.801701 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.801785 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:14.802576 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.803683 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.806803 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.808072 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.808228 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.808299 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.808767 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.808893 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 03:15:14.809168 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.800639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:51995 (local address 127.14.227.130:38285)
0924 03:15:14.800882 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.800890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.801445 (+ 555us) server_negotiation.cc:407] Connection header received
0924 03:15:14.801592 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.801598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.801675 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.801779 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:14.802512 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.803682 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.806804 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.808066 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.808226 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.808299 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.808767 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.808892 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:15:14.809440 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.800609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:38285 (local address 127.14.227.129:51995)
0924 03:15:14.801347 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:14.801363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:14.801458 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:14.801817 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:14.801822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:14.801848 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:14.802332 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.802348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.803847 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.803853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.806655 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.806665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.808333 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.808338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.808570 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.808643 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 03:15:14.808733 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 03:15:14.808945 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":280,"thread_start_us":427,"threads_started":1}
I20260924 03:15:14.809463 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.800470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:45949 (local address 127.14.227.129:52319)
0924 03:15:14.801221 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:14.801255 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:14.801421 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:14.801817 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:14.801822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:14.801911 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:14.802436 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.802449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.803848 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.803853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.806649 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:14.806665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.808314 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.808321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:14.808573 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.808636 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 03:15:14.808733 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:15:14.808945 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":353,"spinlock_wait_cycles":23424,"thread_start_us":444,"threads_started":1}
I20260924 03:15:14.810593 16231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd6eda54d134a2db7a8fd075a48868b" candidate_uuid: "608b3f57a29748888424b20fc28ae2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddc30ad902e462aac94446c564aed28" is_pre_election: true
I20260924 03:15:14.810642 16098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd6eda54d134a2db7a8fd075a48868b" candidate_uuid: "608b3f57a29748888424b20fc28ae2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" is_pre_election: true
I20260924 03:15:14.811198 16098 raft_consensus.cc:2454] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608b3f57a29748888424b20fc28ae2d3 in term 0.
I20260924 03:15:14.811199 16231 raft_consensus.cc:2454] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608b3f57a29748888424b20fc28ae2d3 in term 0.
I20260924 03:15:14.811925 15897 leader_election.cc:302] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 608b3f57a29748888424b20fc28ae2d3, 84d407aa1b454c9b9b6a7dce2df31e4b, 8ddc30ad902e462aac94446c564aed28; no voters:
I20260924 03:15:14.812419 16301 raft_consensus.cc:2781] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:14.812579 16301 raft_consensus.cc:482] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:14.812680 16301 raft_consensus.cc:3037] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:14.815006 16301 raft_consensus.cc:504] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.815577 16301 leader_election.cc:288] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [CANDIDATE]: Term 1 election: Requested vote from peers 8ddc30ad902e462aac94446c564aed28 (127.14.227.131:45949), 84d407aa1b454c9b9b6a7dce2df31e4b (127.14.227.130:38285)
I20260924 03:15:14.816010 16231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd6eda54d134a2db7a8fd075a48868b" candidate_uuid: "608b3f57a29748888424b20fc28ae2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ddc30ad902e462aac94446c564aed28"
I20260924 03:15:14.816023 16098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd6eda54d134a2db7a8fd075a48868b" candidate_uuid: "608b3f57a29748888424b20fc28ae2d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b"
I20260924 03:15:14.816223 16231 raft_consensus.cc:3037] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:14.816234 16098 raft_consensus.cc:3037] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:14.819360 16231 raft_consensus.cc:2454] T acd6eda54d134a2db7a8fd075a48868b P 8ddc30ad902e462aac94446c564aed28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608b3f57a29748888424b20fc28ae2d3 in term 1.
I20260924 03:15:14.819360 16098 raft_consensus.cc:2454] T acd6eda54d134a2db7a8fd075a48868b P 84d407aa1b454c9b9b6a7dce2df31e4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608b3f57a29748888424b20fc28ae2d3 in term 1.
I20260924 03:15:14.819759 15897 leader_election.cc:302] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608b3f57a29748888424b20fc28ae2d3, 8ddc30ad902e462aac94446c564aed28; no voters:
I20260924 03:15:14.820021 16301 raft_consensus.cc:2781] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:14.820457 16301 raft_consensus.cc:686] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [term 1 LEADER]: Becoming Leader. State: Replica: 608b3f57a29748888424b20fc28ae2d3, State: Running, Role: LEADER
I20260924 03:15:14.820984 16301 consensus_queue.cc:237] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } }
I20260924 03:15:14.825183 15820 catalog_manager.cc:5495] T acd6eda54d134a2db7a8fd075a48868b P 608b3f57a29748888424b20fc28ae2d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 608b3f57a29748888424b20fc28ae2d3 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "608b3f57a29748888424b20fc28ae2d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608b3f57a29748888424b20fc28ae2d3" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ddc30ad902e462aac94446c564aed28" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 45949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d407aa1b454c9b9b6a7dce2df31e4b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38285 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:15:14 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:14.886610 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.872347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.14.227.129:33229)
0924 03:15:14.872526 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.872530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.872996 (+ 466us) server_negotiation.cc:407] Connection header received
0924 03:15:14.873216 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.873220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.873299 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.873467 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:14.874942 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.875640 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.876864 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.877266 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.877393 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.881329 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:14.881343 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:14.881358 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:14.881388 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:14.884006 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.884657 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.884665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.884668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.884797 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.885151 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.885157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.885160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.885480 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:14.885794 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.886037 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.886303 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 03:15:14 dist-test-slave-vd0p krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:15:14 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219714, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:14.930007 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.916485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.14.227.129:33229)
0924 03:15:14.916765 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.916773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.917060 (+ 287us) server_negotiation.cc:407] Connection header received
0924 03:15:14.917272 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.917277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.917337 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.917454 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:14.918927 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.919671 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.920907 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.921312 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.921452 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.925160 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:14.925165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:14.925168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:14.925201 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:14.927486 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.928052 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.928059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.928062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.928191 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.928563 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.928569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.928572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.928827 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:14.929099 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.929349 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.929561 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
Sep 24 03:15:14 dist-test-slave-vd0p krb5kdc[15775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:15:14 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:14.972206 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.958724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.14.227.129:33229)
0924 03:15:14.958943 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.958950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.959076 (+ 126us) server_negotiation.cc:407] Connection header received
0924 03:15:14.959233 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.959237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.959312 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.959389 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:14.960870 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.961637 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.962941 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.963401 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.963529 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.967247 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:14.967252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:14.967255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:14.967286 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:14.969690 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.970362 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.970373 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.970382 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.970507 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.970874 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.970878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.970880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.971125 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:14.971373 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.971657 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.971883 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 03:15:14.973802 15924 tablet_service.cc:2908] Rejecting scan request for tablet acd6eda54d134a2db7a8fd075a48868b: Uninitialized: clean time has not yet been initialized
I20260924 03:15:14.976574 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 15881
I20260924 03:15:14.987828 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:14.977488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.14.227.129:33229)
0924 03:15:14.977687 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:14.977696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:14.978006 (+ 310us) server_negotiation.cc:407] Connection header received
0924 03:15:14.978129 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:14.978132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:14.978192 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:14.978268 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:14.979316 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.979955 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.981050 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:14.981452 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:14.981535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:14.983582 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:14.983586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:14.983589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:14.983615 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:14.985756 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.986347 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.986350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.986352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.986467 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:14.986752 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:14.986755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:14.986757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:14.986963 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:14.987122 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:14.987334 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 03:15:14.987474 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:15:14.988379 16004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:15.077786 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 15881
I20260924 03:15:15.093972 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16015
Sep 24 03:15:15 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:15:15.107583 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:15.094751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.14.227.130:38285)
0924 03:15:15.094938 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:15.094942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:15.094960 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:15:15.095037 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:15.095040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:15.095100 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:15.095181 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:15.096412 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:15.097155 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:15.098387 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:15.098773 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:15.098859 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:15.102615 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:15.102627 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:15.102642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:15.102675 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:15.105427 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:15.106019 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:15.106026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:15.106028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:15.106153 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:15.106460 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:15.106462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:15.106464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:15.106796 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:15.106999 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:15.107193 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 03:15:15.107342 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:15:15.108261 16138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:15.194833 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 16015
I20260924 03:15:15.210377 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16148
Sep 24 03:15:15 dist-test-slave-vd0p krb5kdc[15775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:15:15.224166 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:15.211067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.14.227.131:45949)
0924 03:15:15.211241 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:15.211246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:15.211263 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:15:15.211417 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:15.211420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:15.211488 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:15.211570 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:15.212737 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:15.213641 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:15.214891 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:15.215257 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:15.215349 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:15.219085 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:15.219099 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:15.219114 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:15.219142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:15.221754 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:15.222443 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:15.222450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:15.222452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:15.222578 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:15.222933 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:15.222936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:15.222938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:15.223250 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:15.223445 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:15.223745 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 03:15:15.223902 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:15:15.224757 16271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:15.308808 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 16148
I20260924 03:15:15.324719 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 15788
I20260924 03:15:15.325855 15850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:15.435560 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 15788
2026-09-24T03:15:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4216 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:15 dist-test-slave-vd0p krb5kdc[16324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:15 dist-test-slave-vd0p krb5kdc[16324](info): set up 2 sockets
Sep 24 03:15:15 dist-test-slave-vd0p krb5kdc[16324](info): commencing operation
krb5kdc: starting...
W20260924 03:15:16.502089 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:16 dist-test-slave-vd0p krb5kdc[16324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:16.673628 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:34525
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:34525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:17.002949 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:17.003247 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:17.003439 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:17.010744 16337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:17.010871 16337 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:15:17.010960 16337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:17.011065 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:17.011149 16337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:17.011246 16337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:17.021200 16337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:34525
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:34525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:17.022500 16337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:17.023852 16337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:17.035650 16345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:17.035825 16342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:17.035948 16343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:17.036772 16337 server_base.cc:1034] running on GCE node
I20260924 03:15:17.037416 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:17.038766 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:17.039973 16337 hybrid_clock.cc:648] HybridClock initialized: now 1790219717039921 us; error 24 us; skew 500 ppm
I20260924 03:15:17.040235 16337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:17 dist-test-slave-vd0p krb5kdc[16324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219717, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:17.043200 16337 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:17.044577 16337 webserver.cc:460] Webserver started at http://127.14.227.190:33637/ using document root <none> and password file <none>
I20260924 03:15:17.045241 16337 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:17.045346 16337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:17.045656 16337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:17.048440 16337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "a1d7f7f6f48444b8a14c1ef265750b2c"
format_stamp: "Formatted at 2026-09-24 03:15:17 on dist-test-slave-vd0p"
server_key: "0e7116beddf32539479c5e29f035f7d1"
server_key_iv: "e9b4a33b10c59e6addb86158a0817f48"
server_key_version: "encryptionkey@0"
I20260924 03:15:17.049182 16337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "a1d7f7f6f48444b8a14c1ef265750b2c"
format_stamp: "Formatted at 2026-09-24 03:15:17 on dist-test-slave-vd0p"
server_key: "0e7116beddf32539479c5e29f035f7d1"
server_key_iv: "e9b4a33b10c59e6addb86158a0817f48"
server_key_version: "encryptionkey@0"
I20260924 03:15:17.053882 16337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 03:15:17.057119 16352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:17.058399 16337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:17.058566 16337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "a1d7f7f6f48444b8a14c1ef265750b2c"
format_stamp: "Formatted at 2026-09-24 03:15:17 on dist-test-slave-vd0p"
server_key: "0e7116beddf32539479c5e29f035f7d1"
server_key_iv: "e9b4a33b10c59e6addb86158a0817f48"
server_key_version: "encryptionkey@0"
I20260924 03:15:17.058764 16337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:17.080029 16337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:17.083865 16337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:17.084120 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:17.104442 16337 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:34525
I20260924 03:15:17.104481 16413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:34525 every 8 connection(s)
I20260924 03:15:17.105775 16337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:17.109138 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 16337
I20260924 03:15:17.109370 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:17.109421 16414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:17.110080 15246 external_mini_cluster.cc:1428] Setting key 245b3c94f7d90f136db67403da1fddfb
I20260924 03:15:17.122346 16414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c: Bootstrap starting.
Sep 24 03:15:17 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:17.125806 16414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:17.126997 16414 log.cc:824] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:17.129783 16414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c: No bootstrap required, opened a new log
I20260924 03:15:17.132594 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:17.111946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.14.227.190:34525)
0924 03:15:17.112733 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:17.112744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:17.112788 (+ 44us) server_negotiation.cc:407] Connection header received
0924 03:15:17.114924 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:17.114945 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:17.115323 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:17.115643 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:17.117249 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.118588 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.120301 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.120919 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.121068 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:17.125658 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:17.125666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:17.125680 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:17.125721 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:17.128765 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:17.129412 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:17.129418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:17.129423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:17.129589 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:17.129940 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:17.129975 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:17.129978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:17.130554 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:17.130852 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:17.131257 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 03:15:17.131658 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":360,"threads_started":1}
I20260924 03:15:17.138617 16414 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d7f7f6f48444b8a14c1ef265750b2c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34525 } }
I20260924 03:15:17.138970 16414 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:17.139102 16414 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d7f7f6f48444b8a14c1ef265750b2c, State: Initialized, Role: FOLLOWER
I20260924 03:15:17.139744 16414 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d7f7f6f48444b8a14c1ef265750b2c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34525 } }
I20260924 03:15:17.140005 16414 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:17.140156 16414 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:17.140337 16414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:17.142941 16414 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d7f7f6f48444b8a14c1ef265750b2c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34525 } }
I20260924 03:15:17.143486 16414 leader_election.cc:302] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1d7f7f6f48444b8a14c1ef265750b2c; no voters:
I20260924 03:15:17.144060 16414 leader_election.cc:288] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:17.144178 16419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:17.144909 16419 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [term 1 LEADER]: Becoming Leader. State: Replica: a1d7f7f6f48444b8a14c1ef265750b2c, State: Running, Role: LEADER
I20260924 03:15:17.145470 16419 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d7f7f6f48444b8a14c1ef265750b2c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34525 } }
I20260924 03:15:17.146224 16414 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:17.148145 16420 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1d7f7f6f48444b8a14c1ef265750b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d7f7f6f48444b8a14c1ef265750b2c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34525 } } }
I20260924 03:15:17.148285 16421 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1d7f7f6f48444b8a14c1ef265750b2c. Latest consensus state: current_term: 1 leader_uuid: "a1d7f7f6f48444b8a14c1ef265750b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d7f7f6f48444b8a14c1ef265750b2c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34525 } } }
I20260924 03:15:17.148550 16421 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:17.148550 16420 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:17.150990 16425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:17.156658 16425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:17.165284 16425 catalog_manager.cc:1347] Generated new cluster ID: d01dc4cf94f44b96949b6d245c6b381b
I20260924 03:15:17.165390 16425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:17.179315 16425 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:17.180613 16425 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:17.190603 16425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c: Generated new TSK 0
I20260924 03:15:17.191385 16425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:17.237689 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34525
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:17.569599 16440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:17.569873 16440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:17.570056 16440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:17.577234 16440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:17.577365 16440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:17.577539 16440 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:15:17.577680 16440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:17.587702 16440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:34525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:17.589027 16440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:17.590500 16440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:17.604800 16446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:17.604768 16448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:17.605357 16445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:17.606007 16440 server_base.cc:1034] running on GCE node
I20260924 03:15:17.606690 16440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:17.607656 16440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:17.608906 16440 hybrid_clock.cc:648] HybridClock initialized: now 1790219717608826 us; error 59 us; skew 500 ppm
I20260924 03:15:17.609233 16440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:17 dist-test-slave-vd0p krb5kdc[16324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219717, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:17.612509 16440 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:17.614015 16440 webserver.cc:460] Webserver started at http://127.14.227.129:44857/ using document root <none> and password file <none>
I20260924 03:15:17.614724 16440 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:17.614850 16440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:17.615170 16440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:17.618073 16440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "dfa47051a69b4368835400db5698af7f"
format_stamp: "Formatted at 2026-09-24 03:15:17 on dist-test-slave-vd0p"
server_key: "b30b9f1b983176720004984242608df5"
server_key_iv: "45be99bc646337d4dfead4c8fd09a4ee"
server_key_version: "encryptionkey@0"
I20260924 03:15:17.618876 16440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "dfa47051a69b4368835400db5698af7f"
format_stamp: "Formatted at 2026-09-24 03:15:17 on dist-test-slave-vd0p"
server_key: "b30b9f1b983176720004984242608df5"
server_key_iv: "45be99bc646337d4dfead4c8fd09a4ee"
server_key_version: "encryptionkey@0"
I20260924 03:15:17.623799 16440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:15:17.627169 16455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:17.628505 16440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:15:17.628680 16440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "dfa47051a69b4368835400db5698af7f"
format_stamp: "Formatted at 2026-09-24 03:15:17 on dist-test-slave-vd0p"
server_key: "b30b9f1b983176720004984242608df5"
server_key_iv: "45be99bc646337d4dfead4c8fd09a4ee"
server_key_version: "encryptionkey@0"
I20260924 03:15:17.628903 16440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:17.653270 16440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:17.657547 16440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:17.657812 16440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:17.660518 16440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:17.660621 16440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:17.660761 16440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:17.660833 16440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:15:17 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219717, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:17.683398 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:17.661294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:46369 (local address 127.14.227.190:34525)
0924 03:15:17.661785 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:17.661792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:17.663192 (+ 1400us) server_negotiation.cc:407] Connection header received
0924 03:15:17.666431 (+ 3239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:17.666439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:17.666534 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:17.666651 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:17.668883 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.669622 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.671364 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.671784 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.671914 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:17.676214 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:17.676223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:17.676228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:17.676270 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:17.680018 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:17.680743 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:17.680751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:17.680755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:17.680926 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:17.681427 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:17.681432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:17.681435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:17.681705 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:17.681852 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:17.682799 (+ 947us) server_negotiation.cc:299] Negotiation successful
0924 03:15:17.683016 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":253,"threads_started":1}
I20260924 03:15:17.684489 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:17.662061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34525 (local address 127.14.227.129:46369)
0924 03:15:17.663014 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:17.663091 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:17.666039 (+ 2948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:17.666973 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:17.666988 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:17.667789 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:17.668589 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:17.668634 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.669783 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.669786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:17.671218 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:17.671230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.672003 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.672013 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:17.672223 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:17.672953 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:17.672974 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:17.675964 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:17.680155 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:17.680164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:17.680181 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:17.680614 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:17.681072 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:17.681078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:17.681086 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:17.681303 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:17.681847 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:17.681855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:17.682343 (+ 488us) client_negotiation.cc:769] Sending connection context
0924 03:15:17.682709 (+ 366us) client_negotiation.cc:240] Negotiation successful
0924 03:15:17.683283 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":331,"threads_started":1}
I20260924 03:15:17.704114 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:17.691014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:34499 (local address 127.14.227.190:34525)
0924 03:15:17.691156 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:17.691160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:17.691814 (+ 654us) server_negotiation.cc:407] Connection header received
0924 03:15:17.692000 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:17.692004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:17.692062 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:17.692158 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:17.693151 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.693853 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.695695 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.696078 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.696214 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:17.698044 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:17.698059 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:17.698063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:17.698091 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:17.701125 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:17.701887 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:17.701890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:17.701892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:17.702058 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:17.702491 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:17.702496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:17.702500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:17.702913 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:17.703097 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:17.703555 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 03:15:17.703765 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:15:17.704222 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:17.691141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34525 (local address 127.14.227.129:34499)
0924 03:15:17.691712 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:17.691732 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:17.691882 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:17.692180 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:17.692184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:17.692754 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:17.693021 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:17.693030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.694013 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.694019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:17.695553 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:17.695564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:17.696259 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:17.696265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:17.696464 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:17.697140 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:17.697155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:17.697854 (+ 699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:17.701296 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:17.701302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:17.701306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:17.701759 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:17.702163 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:17.702167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:17.702171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:17.702344 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:17.703068 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:17.703074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:17.703291 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 03:15:17.703454 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 03:15:17.703786 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":329,"threads_started":1}
I20260924 03:15:17.716655 16440 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:35625
I20260924 03:15:17.716678 16575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:35625 every 8 connection(s)
I20260924 03:15:17.717904 16440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:17.719955 16576 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34525
I20260924 03:15:17.720194 16576 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:17.720825 16576 heartbeater.cc:511] Master 127.14.227.190:34525 requested a full tablet report, sending...
I20260924 03:15:17.722745 16369 ts_manager.cc:194] Registered new tserver with Master: dfa47051a69b4368835400db5698af7f (127.14.227.129:35625)
I20260924 03:15:17.724155 16369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:46369
I20260924 03:15:17.725835 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 16440
I20260924 03:15:17.726083 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:17.726699 15246 external_mini_cluster.cc:1428] Setting key 9921b531b21b5c582a2eb268684aa7df
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:17.763273 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34525
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:18.072597 16580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:18.072916 16580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:18.073089 16580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:18.080384 16580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:18.080542 16580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:18.080694 16580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:15:18.080832 16580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:18.090739 16580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:34525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:18.092015 16580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:18.093354 16580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:18.106169 16586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:18.106267 16588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:18.106670 16585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:18.110052 16580 server_base.cc:1034] running on GCE node
I20260924 03:15:18.110607 16580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:18.111486 16580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:18.112715 16580 hybrid_clock.cc:648] HybridClock initialized: now 1790219718112647 us; error 47 us; skew 500 ppm
I20260924 03:15:18.112998 16580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:18 dist-test-slave-vd0p krb5kdc[16324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:18.120671 16580 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:18.122128 16580 webserver.cc:460] Webserver started at http://127.14.227.130:40169/ using document root <none> and password file <none>
I20260924 03:15:18.122807 16580 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:18.122929 16580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:18.123234 16580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:18.126097 16580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "9db65ca094ab450daa6169167c78bc4c"
format_stamp: "Formatted at 2026-09-24 03:15:18 on dist-test-slave-vd0p"
server_key: "64b77bb15e415bf602e0ab73c8e01566"
server_key_iv: "f4d5c94955d9bd92e8afe71e0f213813"
server_key_version: "encryptionkey@0"
I20260924 03:15:18.126868 16580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "9db65ca094ab450daa6169167c78bc4c"
format_stamp: "Formatted at 2026-09-24 03:15:18 on dist-test-slave-vd0p"
server_key: "64b77bb15e415bf602e0ab73c8e01566"
server_key_iv: "f4d5c94955d9bd92e8afe71e0f213813"
server_key_version: "encryptionkey@0"
I20260924 03:15:18.131779 16580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 03:15:18.135221 16595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:18.136446 16580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:15:18.136610 16580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "9db65ca094ab450daa6169167c78bc4c"
format_stamp: "Formatted at 2026-09-24 03:15:18 on dist-test-slave-vd0p"
server_key: "64b77bb15e415bf602e0ab73c8e01566"
server_key_iv: "f4d5c94955d9bd92e8afe71e0f213813"
server_key_version: "encryptionkey@0"
I20260924 03:15:18.136827 16580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:18.150907 16580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:18.155238 16580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:18.155511 16580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:18.158097 16580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:18.158205 16580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:18.158388 16580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:18.158494 16580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:15:18 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:18.185093 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.159323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34525 (local address 127.14.227.130:37355)
0924 03:15:18.160603 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.160686 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.163477 (+ 2791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.164378 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.164394 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.165292 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:18.166276 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.166307 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.167649 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.167661 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.170689 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.170709 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.171913 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.171917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.172107 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.172834 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:18.172875 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:18.176214 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:18.179770 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:18.179786 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:18.179808 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:18.180597 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:18.181039 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:18.181044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:18.181047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:18.181373 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:18.181903 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:18.181911 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:18.182472 (+ 561us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.182803 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.183379 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":728,"threads_started":1}
I20260924 03:15:18.186213 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.160124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:37355 (local address 127.14.227.190:34525)
0924 03:15:18.161255 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.161263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.161288 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:15:18.163827 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.163835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.163958 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.164074 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:18.166475 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.167483 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.170863 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.171516 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.171646 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.176463 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:18.176468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:18.176472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:18.176533 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:18.179597 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.180758 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.180765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.180771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.180924 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.181492 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.181497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.181500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.181773 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:18.181923 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.185649 (+ 3726us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.185897 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961}
I20260924 03:15:18.207525 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.192228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:39401 (local address 127.14.227.190:34525)
0924 03:15:18.192429 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.192437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.193141 (+ 704us) server_negotiation.cc:407] Connection header received
0924 03:15:18.193335 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.193338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.193395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.193472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:18.195040 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.195976 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.198029 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.198616 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.198698 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.200497 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:18.200506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:18.200513 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:18.200563 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:18.204450 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.205534 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.205542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.205547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.205715 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.206191 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.206195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.206197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.206512 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:18.206629 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.207157 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.207314 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:15:18.208168 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.192306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34525 (local address 127.14.227.130:39401)
0924 03:15:18.193006 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.193048 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.193212 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.193531 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.193539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.194299 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:18.194861 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.194881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.196100 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.196105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.197801 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.197819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.198878 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.198882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.198986 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.199576 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:18.199592 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:18.200293 (+ 701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:18.204621 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:18.204632 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:18.204639 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:18.205404 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:18.205846 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:18.205853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:18.205860 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:18.206071 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:18.206629 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:18.206636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:18.206892 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.207097 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.207525 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":345,"threads_started":1}
I20260924 03:15:18.215029 16580 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:45373
I20260924 03:15:18.215047 16714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:45373 every 8 connection(s)
I20260924 03:15:18.216142 16580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:18.217862 16715 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34525
I20260924 03:15:18.218101 16715 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:18.218691 16715 heartbeater.cc:511] Master 127.14.227.190:34525 requested a full tablet report, sending...
I20260924 03:15:18.219215 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 16580
I20260924 03:15:18.219419 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:18.219909 15246 external_mini_cluster.cc:1428] Setting key 4e9d519b746b71dc28ca8159e2ca3f4c
I20260924 03:15:18.219975 16369 ts_manager.cc:194] Registered new tserver with Master: 9db65ca094ab450daa6169167c78bc4c (127.14.227.130:45373)
I20260924 03:15:18.220865 16369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:37355
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:18.256901 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34525
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:18.569996 16719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:18.570307 16719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:18.570484 16719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:18.577477 16719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:18.577571 16719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:18.577682 16719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:15:18.577786 16719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:18.587198 16719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:34525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:18.588537 16719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:18.589973 16719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:18.602494 16724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:18.602560 16725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:18.603091 16727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:18.603960 16719 server_base.cc:1034] running on GCE node
I20260924 03:15:18.604561 16719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:18.605388 16719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:18.606621 16719 hybrid_clock.cc:648] HybridClock initialized: now 1790219718606553 us; error 43 us; skew 500 ppm
I20260924 03:15:18.606958 16719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:18 dist-test-slave-vd0p krb5kdc[16324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:18.609992 16719 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:18.611395 16719 webserver.cc:460] Webserver started at http://127.14.227.131:45517/ using document root <none> and password file <none>
I20260924 03:15:18.612069 16719 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:18.612200 16719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:18.612509 16719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:18.615340 16719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "6d5c396071f745edb41c02b3598a9084"
format_stamp: "Formatted at 2026-09-24 03:15:18 on dist-test-slave-vd0p"
server_key: "47c89b02824f51fad1b61856e236709d"
server_key_iv: "89987472aab54fe910e037032fac97f1"
server_key_version: "encryptionkey@0"
I20260924 03:15:18.616086 16719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "6d5c396071f745edb41c02b3598a9084"
format_stamp: "Formatted at 2026-09-24 03:15:18 on dist-test-slave-vd0p"
server_key: "47c89b02824f51fad1b61856e236709d"
server_key_iv: "89987472aab54fe910e037032fac97f1"
server_key_version: "encryptionkey@0"
I20260924 03:15:18.620923 16719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:15:18.624333 16734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:18.625696 16719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:15:18.625856 16719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "6d5c396071f745edb41c02b3598a9084"
format_stamp: "Formatted at 2026-09-24 03:15:18 on dist-test-slave-vd0p"
server_key: "47c89b02824f51fad1b61856e236709d"
server_key_iv: "89987472aab54fe910e037032fac97f1"
server_key_version: "encryptionkey@0"
I20260924 03:15:18.626117 16719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:18.645061 16719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:18.649242 16719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:18.649515 16719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:18.652263 16719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:18.652374 16719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:18.652514 16719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:18.652598 16719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:15:18 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:18.677320 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.652841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:57207 (local address 127.14.227.190:34525)
0924 03:15:18.653069 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.653075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.654516 (+ 1441us) server_negotiation.cc:407] Connection header received
0924 03:15:18.656636 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.656643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.656749 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.656884 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:18.659969 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.660914 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.664376 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.664931 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.665080 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.670239 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:18.670245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:18.670248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:18.670284 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:18.673307 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.674424 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.674433 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.674440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.674641 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.675075 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.675082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.675089 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.675553 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:18.675722 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.676457 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.676786 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 03:15:18.677747 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.653462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34525 (local address 127.14.227.131:57207)
0924 03:15:18.654340 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.654388 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.656270 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.657411 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.657427 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.658566 (+ 1139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:18.659734 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.659771 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.661149 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.661155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.664194 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.664216 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.665267 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.665275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.665655 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.666619 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:18.666664 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:18.669962 (+ 3298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:18.673462 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:18.673474 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:18.673501 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:18.674236 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:18.674763 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:18.674766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:18.674769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:18.674960 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:18.675689 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:18.675695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:18.676032 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.676244 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.676631 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":406,"threads_started":1}
I20260924 03:15:18.698258 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.683713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:55825 (local address 127.14.227.190:34525)
0924 03:15:18.683932 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.683940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.684644 (+ 704us) server_negotiation.cc:407] Connection header received
0924 03:15:18.684895 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.684901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.684965 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.685061 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:18.686422 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.687518 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.689119 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.689552 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.689694 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.692404 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:18.692413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:18.692418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:18.692461 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:18.695437 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.696403 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.696407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.696409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.696522 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.696874 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.696880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.696883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.697125 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:18.697287 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.697657 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.697875 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:15:18.698374 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.683742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34525 (local address 127.14.227.131:55825)
0924 03:15:18.684515 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.684545 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.684710 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.685089 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.685093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.685730 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:18.686288 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.686299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.687661 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.687666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.688986 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.688998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.689729 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.689736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.689904 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.690639 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:18.690688 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:18.692176 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:18.695616 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:18.695624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:18.695629 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:18.696233 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:18.696628 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:18.696637 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:18.696640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:18.696765 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:18.697246 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:18.697250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:18.697423 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.697552 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.697877 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":410,"threads_started":1}
I20260924 03:15:18.707062 16719 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:35071
I20260924 03:15:18.707082 16853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:35071 every 8 connection(s)
I20260924 03:15:18.708258 16719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:18.710356 16854 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34525
I20260924 03:15:18.710568 16854 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:18.711186 16854 heartbeater.cc:511] Master 127.14.227.190:34525 requested a full tablet report, sending...
I20260924 03:15:18.712448 16368 ts_manager.cc:194] Registered new tserver with Master: 6d5c396071f745edb41c02b3598a9084 (127.14.227.131:35071)
I20260924 03:15:18.713294 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 16719
I20260924 03:15:18.713474 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:18.713562 16368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:57207
I20260924 03:15:18.714022 15246 external_mini_cluster.cc:1428] Setting key 6de2b128a8657bd0fb9c327cc81c5ab7
I20260924 03:15:18.717018 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:18.726824 16576 heartbeater.cc:503] Master 127.14.227.190:34525 was elected leader, sending a full tablet report...
Sep 24 03:15:18 dist-test-slave-vd0p krb5kdc[16324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:15:18 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:18.760759 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.747925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.14.227.190:34525)
0924 03:15:18.748143 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.748148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.748258 (+ 110us) server_negotiation.cc:407] Connection header received
0924 03:15:18.748412 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.748416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.748479 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.748566 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:18.749932 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.750683 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.751916 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.752311 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.752402 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.756110 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:18.756115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:18.756118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:18.756148 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:18.758714 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.759244 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.759248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.759250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.759366 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:18.759664 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:18.759667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:18.759669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:18.759945 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:18.760118 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.760337 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.760490 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:15:18.764595 16368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:18.771281 16368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:18.795054 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.782858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.14.227.130:45373)
0924 03:15:18.784051 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.784061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.784086 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:15:18.784186 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.784196 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.784455 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.784712 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:18.785418 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.787436 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.791250 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.793132 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.793307 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.793526 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.794418 (+ 892us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.794652 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":920,"threads_started":1}
I20260924 03:15:18.796118 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.782736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45373 (local address 127.0.0.1:46004)
0924 03:15:18.783393 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.783439 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.783587 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.784703 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.784713 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.784744 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:18.785187 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.785201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.787735 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.787743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.791016 (+ 3273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.791037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.793773 (+ 2736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.793782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.794084 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.794155 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.795309 (+ 1154us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.795533 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":43,"thread_start_us":281,"threads_started":1}
I20260924 03:15:18.799059 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.783837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:35625 (local address 127.0.0.1:38854)
0924 03:15:18.784364 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.784380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.784481 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.787450 (+ 2969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.787454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.787484 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:18.788134 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.788155 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.791835 (+ 3680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.791843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.795876 (+ 4033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.795895 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.798298 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.798306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.798467 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.798497 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.798582 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.798696 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":4,"thread_start_us":286,"threads_started":1}
I20260924 03:15:18.799805 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.784292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.14.227.129:35625)
0924 03:15:18.785347 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.785358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.785388 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:15:18.785493 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.785499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.786986 (+ 1487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.787225 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:18.788370 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.791618 (+ 3248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.796101 (+ 4483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.797783 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.797973 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.798132 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.799372 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.799536 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":780,"threads_started":1}
I20260924 03:15:18.800215 16645 tablet_service.cc:1467] Processing CreateTablet for tablet d62bde6490b74236b82ce10fb9a16893 (DEFAULT_TABLE table=test-table [id=4a9bcdc2b52549ddb3029235a8210524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:18.801534 16645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d62bde6490b74236b82ce10fb9a16893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:18.803623 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.784985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:35071 (local address 127.0.0.1:45758)
0924 03:15:18.796366 (+ 11381us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.796382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.796475 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.797923 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.797927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.797975 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:18.798136 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.798146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.799906 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.799914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.801686 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.801706 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.802912 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.802921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.803091 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.803120 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.803231 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.803358 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11281,"thread_start_us":11641,"threads_started":1}
I20260924 03:15:18.803321 16506 tablet_service.cc:1467] Processing CreateTablet for tablet d62bde6490b74236b82ce10fb9a16893 (DEFAULT_TABLE table=test-table [id=4a9bcdc2b52549ddb3029235a8210524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:18.803977 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.785552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.14.227.131:35071)
0924 03:15:18.797482 (+ 11930us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.797488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.797513 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:15:18.797587 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.797592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.797744 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.797937 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:18.798366 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.799775 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.801840 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.802762 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.802963 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.803118 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.803277 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.803427 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11798,"thread_start_us":12623,"threads_started":1}
I20260924 03:15:18.804590 16506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d62bde6490b74236b82ce10fb9a16893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:18.806779 16784 tablet_service.cc:1467] Processing CreateTablet for tablet d62bde6490b74236b82ce10fb9a16893 (DEFAULT_TABLE table=test-table [id=4a9bcdc2b52549ddb3029235a8210524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:18.807770 16784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d62bde6490b74236b82ce10fb9a16893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:18.815666 16873 tablet_bootstrap.cc:492] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: Bootstrap starting.
I20260924 03:15:18.817963 16874 tablet_bootstrap.cc:492] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: Bootstrap starting.
I20260924 03:15:18.819324 16875 tablet_bootstrap.cc:492] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: Bootstrap starting.
I20260924 03:15:18.819366 16873 tablet_bootstrap.cc:654] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:18.821228 16874 tablet_bootstrap.cc:654] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:18.821283 16873 log.cc:824] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:18.822449 16874 log.cc:824] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:18.822556 16875 tablet_bootstrap.cc:654] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:18.823523 16875 log.cc:824] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:18.824090 16873 tablet_bootstrap.cc:492] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: No bootstrap required, opened a new log
I20260924 03:15:18.824353 16873 ts_tablet_manager.cc:1397] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 03:15:18.825251 16874 tablet_bootstrap.cc:492] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: No bootstrap required, opened a new log
I20260924 03:15:18.825502 16874 ts_tablet_manager.cc:1397] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 03:15:18.826345 16875 tablet_bootstrap.cc:492] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: No bootstrap required, opened a new log
I20260924 03:15:18.826581 16875 ts_tablet_manager.cc:1397] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 03:15:18.832127 16873 raft_consensus.cc:348] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.832464 16873 raft_consensus.cc:374] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:18.832576 16873 raft_consensus.cc:729] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db65ca094ab450daa6169167c78bc4c, State: Initialized, Role: FOLLOWER
I20260924 03:15:18.833153 16873 consensus_queue.cc:260] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.834175 16715 heartbeater.cc:503] Master 127.14.227.190:34525 was elected leader, sending a full tablet report...
I20260924 03:15:18.834779 16873 ts_tablet_manager.cc:1428] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260924 03:15:18.836315 16874 raft_consensus.cc:348] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.836699 16874 raft_consensus.cc:374] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:18.836814 16874 raft_consensus.cc:729] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa47051a69b4368835400db5698af7f, State: Initialized, Role: FOLLOWER
I20260924 03:15:18.837023 16875 raft_consensus.cc:348] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.837410 16875 raft_consensus.cc:374] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:18.837495 16875 raft_consensus.cc:729] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5c396071f745edb41c02b3598a9084, State: Initialized, Role: FOLLOWER
I20260924 03:15:18.837388 16874 consensus_queue.cc:260] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.838137 16875 consensus_queue.cc:260] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.839210 16854 heartbeater.cc:503] Master 127.14.227.190:34525 was elected leader, sending a full tablet report...
I20260924 03:15:18.839670 16875 ts_tablet_manager.cc:1428] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 03:15:18.840131 16874 ts_tablet_manager.cc:1428] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 03:15:18.860502 16879 raft_consensus.cc:482] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:18.860752 16879 raft_consensus.cc:504] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.862398 16879 leader_election.cc:288] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa47051a69b4368835400db5698af7f (127.14.227.129:35625), 6d5c396071f745edb41c02b3598a9084 (127.14.227.131:35071)
I20260924 03:15:18.868355 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.862720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:43961 (local address 127.14.227.129:35625)
0924 03:15:18.862904 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.862908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.863538 (+ 630us) server_negotiation.cc:407] Connection header received
0924 03:15:18.863617 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.863622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.863696 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.863806 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:18.864322 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.865023 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.866746 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.867471 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.867636 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.867712 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.867989 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.868110 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:15:18.868409 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.862606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:35625 (local address 127.14.227.130:43961)
0924 03:15:18.863215 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.863234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.863374 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.863843 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.863849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.863875 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:18.864185 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.864193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.865142 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.865150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.866603 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.866614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.867686 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.867693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.867836 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.867859 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.867956 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.868086 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":304,"threads_started":1}
I20260924 03:15:18.868724 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.863530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:42233 (local address 127.14.227.131:35071)
0924 03:15:18.863720 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.863726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.863938 (+ 212us) server_negotiation.cc:407] Connection header received
0924 03:15:18.864140 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.864146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.864204 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.864293 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:18.864806 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.865469 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.867129 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.867863 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.868040 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.868106 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.868375 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.868490 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:15:18.868788 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.863294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:35071 (local address 127.14.227.130:42233)
0924 03:15:18.863849 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:18.863862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:18.864017 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:18.864322 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:18.864326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:18.864351 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:18.864653 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.864661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.865605 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.865610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.866987 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:18.866998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.868065 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.868071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:18.868221 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.868244 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:15:18.868343 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:15:18.868469 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":266,"threads_started":1}
I20260924 03:15:18.869249 16531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d62bde6490b74236b82ce10fb9a16893" candidate_uuid: "9db65ca094ab450daa6169167c78bc4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa47051a69b4368835400db5698af7f" is_pre_election: true
I20260924 03:15:18.869613 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d62bde6490b74236b82ce10fb9a16893" candidate_uuid: "9db65ca094ab450daa6169167c78bc4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5c396071f745edb41c02b3598a9084" is_pre_election: true
I20260924 03:15:18.869707 16531 raft_consensus.cc:2454] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db65ca094ab450daa6169167c78bc4c in term 0.
I20260924 03:15:18.870057 16809 raft_consensus.cc:2454] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db65ca094ab450daa6169167c78bc4c in term 0.
I20260924 03:15:18.870370 16599 leader_election.cc:302] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9db65ca094ab450daa6169167c78bc4c, dfa47051a69b4368835400db5698af7f; no voters:
I20260924 03:15:18.870759 16879 raft_consensus.cc:2781] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:18.870918 16879 raft_consensus.cc:482] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:18.871032 16879 raft_consensus.cc:3037] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:18.873131 16879 raft_consensus.cc:504] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.873646 16879 leader_election.cc:288] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [CANDIDATE]: Term 1 election: Requested vote from peers dfa47051a69b4368835400db5698af7f (127.14.227.129:35625), 6d5c396071f745edb41c02b3598a9084 (127.14.227.131:35071)
I20260924 03:15:18.874115 16531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d62bde6490b74236b82ce10fb9a16893" candidate_uuid: "9db65ca094ab450daa6169167c78bc4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa47051a69b4368835400db5698af7f"
I20260924 03:15:18.874131 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d62bde6490b74236b82ce10fb9a16893" candidate_uuid: "9db65ca094ab450daa6169167c78bc4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5c396071f745edb41c02b3598a9084"
I20260924 03:15:18.874311 16809 raft_consensus.cc:3037] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:18.874320 16531 raft_consensus.cc:3037] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:18.877416 16809 raft_consensus.cc:2454] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db65ca094ab450daa6169167c78bc4c in term 1.
I20260924 03:15:18.877416 16531 raft_consensus.cc:2454] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db65ca094ab450daa6169167c78bc4c in term 1.
I20260924 03:15:18.877825 16599 leader_election.cc:302] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5c396071f745edb41c02b3598a9084, 9db65ca094ab450daa6169167c78bc4c; no voters:
I20260924 03:15:18.878118 16879 raft_consensus.cc:2781] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:18.878582 16879 raft_consensus.cc:686] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 1 LEADER]: Becoming Leader. State: Replica: 9db65ca094ab450daa6169167c78bc4c, State: Running, Role: LEADER
I20260924 03:15:18.879098 16879 consensus_queue.cc:237] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:18.883623 16368 catalog_manager.cc:5495] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9db65ca094ab450daa6169167c78bc4c (127.14.227.130). New cstate: current_term: 1 leader_uuid: "9db65ca094ab450daa6169167c78bc4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } health_report { overall_health: UNKNOWN } } }
I20260924 03:15:18.918917 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:18.914133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.14.227.130:45373)
0924 03:15:18.914302 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:18.914306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:18.914401 (+ 95us) server_negotiation.cc:407] Connection header received
0924 03:15:18.914656 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:18.914659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:18.914733 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:18.914851 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:18.915141 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.915846 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.917050 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:18.917427 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:18.917512 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:18.917912 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:18.918432 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:18.918534 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:18.918651 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 03:15:18.918712 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 03:15:18.926821 16809 raft_consensus.cc:1261] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Refusing update from remote peer 9db65ca094ab450daa6169167c78bc4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:18.926821 16531 raft_consensus.cc:1261] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Refusing update from remote peer 9db65ca094ab450daa6169167c78bc4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:18.927734 16884 consensus_queue.cc:1035] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:18.928079 16879 consensus_queue.cc:1035] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:18.945780 16889 mvcc.cc:205] Tried to move back new op lower bound from 7332739968715075584 to 7332739968532819968. Current Snapshot: MvccSnapshot[applied={T|T < 7332739968715075584}]
I20260924 03:15:18.962142 16368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:18.962380 16368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:18.965716 16368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c: Sending DeleteTablet for 3 replicas of tablet d62bde6490b74236b82ce10fb9a16893
I20260924 03:15:18.966790 16645 tablet_service.cc:1514] Processing DeleteTablet for tablet d62bde6490b74236b82ce10fb9a16893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:18 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:46004
I20260924 03:15:18.966967 16784 tablet_service.cc:1514] Processing DeleteTablet for tablet d62bde6490b74236b82ce10fb9a16893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:18 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:45758
I20260924 03:15:18.967499 16506 tablet_service.cc:1514] Processing DeleteTablet for tablet d62bde6490b74236b82ce10fb9a16893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:18 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38854
I20260924 03:15:18.969991 16368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:18.970692 16368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:18.973706 16899 tablet_replica.cc:330] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: stopping tablet replica
I20260924 03:15:18.974130 16899 raft_consensus.cc:2229] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:18.974607 16899 raft_consensus.cc:2258] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:18.979966 16783 tablet_service.cc:1467] Processing CreateTablet for tablet c4712627d7874279ae880eaf8c861862 (DEFAULT_TABLE table=test-table [id=d5dd2e3b10eb49819038741ee2012c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:18.980691 16783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4712627d7874279ae880eaf8c861862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:18.983134 16644 tablet_service.cc:1467] Processing CreateTablet for tablet c4712627d7874279ae880eaf8c861862 (DEFAULT_TABLE table=test-table [id=d5dd2e3b10eb49819038741ee2012c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:18.983826 16644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4712627d7874279ae880eaf8c861862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:18.987315 16506 tablet_service.cc:1467] Processing CreateTablet for tablet c4712627d7874279ae880eaf8c861862 (DEFAULT_TABLE table=test-table [id=d5dd2e3b10eb49819038741ee2012c95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:18.987772 16506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4712627d7874279ae880eaf8c861862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:18.989077 16897 tablet_replica.cc:330] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: stopping tablet replica
I20260924 03:15:18.989548 16897 raft_consensus.cc:2229] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:15:18.990235 16897 raft_consensus.cc:2258] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:18.991037 16898 tablet_replica.cc:330] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: stopping tablet replica
I20260924 03:15:18.991513 16898 raft_consensus.cc:2229] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:18.993737 16875 tablet_bootstrap.cc:492] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: Bootstrap starting.
I20260924 03:15:18.998782 16897 ts_tablet_manager.cc:1905] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:19.000949 16874 tablet_bootstrap.cc:492] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: Bootstrap starting.
I20260924 03:15:19.001292 16873 tablet_bootstrap.cc:492] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: Bootstrap starting.
I20260924 03:15:19.002115 16899 ts_tablet_manager.cc:1905] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:19.002533 16874 tablet_bootstrap.cc:654] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:19.002871 16873 tablet_bootstrap.cc:654] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:19.006568 16874 tablet_bootstrap.cc:492] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: No bootstrap required, opened a new log
I20260924 03:15:19.006711 16874 ts_tablet_manager.cc:1397] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 03:15:19.007581 16874 raft_consensus.cc:348] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.007826 16874 raft_consensus.cc:374] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:19.007922 16874 raft_consensus.cc:729] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa47051a69b4368835400db5698af7f, State: Initialized, Role: FOLLOWER
I20260924 03:15:19.008275 16874 consensus_queue.cc:260] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.011605 16899 ts_tablet_manager.cc:1918] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:15:19.011834 16899 log.cc:1196] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/d62bde6490b74236b82ce10fb9a16893
I20260924 03:15:19.012283 16874 ts_tablet_manager.cc:1428] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 03:15:19.012688 16899 ts_tablet_manager.cc:1939] T d62bde6490b74236b82ce10fb9a16893 P dfa47051a69b4368835400db5698af7f: Deleting consensus metadata
I20260924 03:15:19.014720 16898 raft_consensus.cc:2258] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:19.014936 16356 catalog_manager.cc:4841] TS dfa47051a69b4368835400db5698af7f (127.14.227.129:35625): tablet d62bde6490b74236b82ce10fb9a16893 (table test-table [id=4a9bcdc2b52549ddb3029235a8210524]) successfully deleted
I20260924 03:15:19.016235 16873 tablet_bootstrap.cc:492] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: No bootstrap required, opened a new log
I20260924 03:15:19.016398 16873 ts_tablet_manager.cc:1397] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260924 03:15:19.016448 16875 tablet_bootstrap.cc:654] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:19.016822 16898 ts_tablet_manager.cc:1905] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:19.017266 16873 raft_consensus.cc:348] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.017531 16873 raft_consensus.cc:374] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:19.017635 16873 raft_consensus.cc:729] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db65ca094ab450daa6169167c78bc4c, State: Initialized, Role: FOLLOWER
I20260924 03:15:19.017901 16873 consensus_queue.cc:260] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.018515 16873 ts_tablet_manager.cc:1428] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:15:19.018715 16875 tablet_bootstrap.cc:492] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: No bootstrap required, opened a new log
I20260924 03:15:19.018875 16875 ts_tablet_manager.cc:1397] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.004s
I20260924 03:15:19.020401 16875 raft_consensus.cc:348] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.020618 16875 raft_consensus.cc:374] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:19.020800 16875 raft_consensus.cc:729] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5c396071f745edb41c02b3598a9084, State: Initialized, Role: FOLLOWER
I20260924 03:15:19.022132 16897 ts_tablet_manager.cc:1918] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:15:19.022331 16897 log.cc:1196] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/d62bde6490b74236b82ce10fb9a16893
I20260924 03:15:19.022919 16897 ts_tablet_manager.cc:1939] T d62bde6490b74236b82ce10fb9a16893 P 9db65ca094ab450daa6169167c78bc4c: Deleting consensus metadata
I20260924 03:15:19.024601 16355 catalog_manager.cc:4841] TS 9db65ca094ab450daa6169167c78bc4c (127.14.227.130:45373): tablet d62bde6490b74236b82ce10fb9a16893 (table test-table [id=4a9bcdc2b52549ddb3029235a8210524]) successfully deleted
I20260924 03:15:19.024621 16875 consensus_queue.cc:260] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.025722 16875 ts_tablet_manager.cc:1428] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260924 03:15:19.026152 16898 ts_tablet_manager.cc:1918] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:15:19.026304 16898 log.cc:1196] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/d62bde6490b74236b82ce10fb9a16893
I20260924 03:15:19.026767 16898 ts_tablet_manager.cc:1939] T d62bde6490b74236b82ce10fb9a16893 P 6d5c396071f745edb41c02b3598a9084: Deleting consensus metadata
I20260924 03:15:19.028240 16356 catalog_manager.cc:4841] TS 6d5c396071f745edb41c02b3598a9084 (127.14.227.131:35071): tablet d62bde6490b74236b82ce10fb9a16893 (table test-table [id=4a9bcdc2b52549ddb3029235a8210524]) successfully deleted
W20260924 03:15:19.210757 16855 tablet.cc:2369] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:15:19.219389 16716 tablet.cc:2369] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:19.220408 16893 raft_consensus.cc:482] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:19.220608 16893 raft_consensus.cc:504] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
W20260924 03:15:19.221323 16577 tablet.cc:2369] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:19.221493 16893 leader_election.cc:288] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa47051a69b4368835400db5698af7f (127.14.227.129:35625), 6d5c396071f745edb41c02b3598a9084 (127.14.227.131:35071)
I20260924 03:15:19.222289 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4712627d7874279ae880eaf8c861862" candidate_uuid: "9db65ca094ab450daa6169167c78bc4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5c396071f745edb41c02b3598a9084" is_pre_election: true
I20260924 03:15:19.222276 16530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4712627d7874279ae880eaf8c861862" candidate_uuid: "9db65ca094ab450daa6169167c78bc4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa47051a69b4368835400db5698af7f" is_pre_election: true
I20260924 03:15:19.222522 16809 raft_consensus.cc:2454] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db65ca094ab450daa6169167c78bc4c in term 0.
I20260924 03:15:19.222574 16530 raft_consensus.cc:2454] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db65ca094ab450daa6169167c78bc4c in term 0.
I20260924 03:15:19.222972 16599 leader_election.cc:302] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5c396071f745edb41c02b3598a9084, 9db65ca094ab450daa6169167c78bc4c; no voters:
I20260924 03:15:19.223270 16893 raft_consensus.cc:2781] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:19.223407 16893 raft_consensus.cc:482] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:19.223518 16893 raft_consensus.cc:3037] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:19.226944 16893 raft_consensus.cc:504] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.227593 16893 leader_election.cc:288] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [CANDIDATE]: Term 1 election: Requested vote from peers dfa47051a69b4368835400db5698af7f (127.14.227.129:35625), 6d5c396071f745edb41c02b3598a9084 (127.14.227.131:35071)
I20260924 03:15:19.228176 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4712627d7874279ae880eaf8c861862" candidate_uuid: "9db65ca094ab450daa6169167c78bc4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5c396071f745edb41c02b3598a9084"
I20260924 03:15:19.228364 16809 raft_consensus.cc:3037] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:19.230245 16530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4712627d7874279ae880eaf8c861862" candidate_uuid: "9db65ca094ab450daa6169167c78bc4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa47051a69b4368835400db5698af7f"
I20260924 03:15:19.230446 16530 raft_consensus.cc:3037] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:19.231244 16809 raft_consensus.cc:2454] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db65ca094ab450daa6169167c78bc4c in term 1.
I20260924 03:15:19.231876 16599 leader_election.cc:302] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5c396071f745edb41c02b3598a9084, 9db65ca094ab450daa6169167c78bc4c; no voters:
I20260924 03:15:19.232232 16893 raft_consensus.cc:2781] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:19.232482 16893 raft_consensus.cc:686] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 1 LEADER]: Becoming Leader. State: Replica: 9db65ca094ab450daa6169167c78bc4c, State: Running, Role: LEADER
I20260924 03:15:19.232857 16893 consensus_queue.cc:237] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.233830 16530 raft_consensus.cc:2454] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db65ca094ab450daa6169167c78bc4c in term 1.
I20260924 03:15:19.238371 16368 catalog_manager.cc:5495] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9db65ca094ab450daa6169167c78bc4c (127.14.227.130). New cstate: current_term: 1 leader_uuid: "9db65ca094ab450daa6169167c78bc4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:15:19 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219717, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:19.325484 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.312285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34525 (local address 127.0.0.1:45898)
0924 03:15:19.312727 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:19.312751 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:19.312927 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:19.313281 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:19.313285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:19.314037 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:19.314270 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.314280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.315304 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.315309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.316219 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.316230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.316941 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.316946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.317134 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.317806 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:19.317822 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:19.320145 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:19.323031 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:19.323044 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:19.323051 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:19.323446 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:19.323777 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:19.323784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:19.323787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:19.323909 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:19.324397 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:19.324404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:19.324554 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 03:15:19.324722 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 03:15:19.325013 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":1,"thread_start_us":230,"threads_started":1}
I20260924 03:15:19.325518 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.312298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.14.227.190:34525)
0924 03:15:19.313037 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.313042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.313063 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:15:19.313102 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.313105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.313154 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.313260 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:19.314418 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.315175 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.316360 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.316793 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.316925 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.320344 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:19.320348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:19.320351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:19.320379 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:19.322901 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.323566 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.323571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.323578 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.323684 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.324028 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.324036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.324039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.324277 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:19.324415 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.324790 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.325120 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":628,"thread_start_us":512,"threads_started":1}
I20260924 03:15:19.330876 16368 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:45898:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:15:19.331797 16368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:19.338235 16506 tablet_service.cc:1467] Processing CreateTablet for tablet 343f859014a54138bc271b98905d23cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ad0364f20a349169c534d668ed981a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:15:19.338773 16506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343f859014a54138bc271b98905d23cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:19.338635 16644 tablet_service.cc:1467] Processing CreateTablet for tablet 343f859014a54138bc271b98905d23cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ad0364f20a349169c534d668ed981a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:15:19.338620 16783 tablet_service.cc:1467] Processing CreateTablet for tablet 343f859014a54138bc271b98905d23cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ad0364f20a349169c534d668ed981a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:15:19.339087 16644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343f859014a54138bc271b98905d23cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:19.339138 16783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343f859014a54138bc271b98905d23cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:19.346407 16874 tablet_bootstrap.cc:492] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: Bootstrap starting.
I20260924 03:15:19.346895 16873 tablet_bootstrap.cc:492] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: Bootstrap starting.
I20260924 03:15:19.348171 16874 tablet_bootstrap.cc:654] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:19.348711 16873 tablet_bootstrap.cc:654] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:19.349216 16875 tablet_bootstrap.cc:492] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: Bootstrap starting.
I20260924 03:15:19.351220 16875 tablet_bootstrap.cc:654] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:19.355050 16875 tablet_bootstrap.cc:492] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: No bootstrap required, opened a new log
I20260924 03:15:19.355244 16875 ts_tablet_manager.cc:1397] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 03:15:19.355985 16873 tablet_bootstrap.cc:492] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: No bootstrap required, opened a new log
I20260924 03:15:19.355974 16875 raft_consensus.cc:348] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.356148 16873 ts_tablet_manager.cc:1397] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 03:15:19.356245 16875 raft_consensus.cc:374] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:19.356343 16875 raft_consensus.cc:729] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5c396071f745edb41c02b3598a9084, State: Initialized, Role: FOLLOWER
I20260924 03:15:19.356577 16875 consensus_queue.cc:260] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.356937 16873 raft_consensus.cc:348] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.357239 16873 raft_consensus.cc:374] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:19.357046 16881 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.357331 16873 raft_consensus.cc:729] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db65ca094ab450daa6169167c78bc4c, State: Initialized, Role: FOLLOWER
I20260924 03:15:19.357385 16881 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:19.357573 16873 consensus_queue.cc:260] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.358047 16893 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.358315 16893 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:19.358755 16873 ts_tablet_manager.cc:1428] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 03:15:19.360450 16875 ts_tablet_manager.cc:1428] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260924 03:15:19.368654 16874 tablet_bootstrap.cc:492] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: No bootstrap required, opened a new log
I20260924 03:15:19.369127 16874 ts_tablet_manager.cc:1397] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20260924 03:15:19.369815 16874 raft_consensus.cc:348] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.370069 16874 raft_consensus.cc:374] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:19.370157 16874 raft_consensus.cc:729] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa47051a69b4368835400db5698af7f, State: Initialized, Role: FOLLOWER
I20260924 03:15:19.370386 16874 consensus_queue.cc:260] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.370872 16874 ts_tablet_manager.cc:1428] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:15:19.370787 16880 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.371147 16880 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:19.371466 16881 raft_consensus.cc:482] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:19.371626 16881 raft_consensus.cc:504] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.373296 16881 leader_election.cc:288] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa47051a69b4368835400db5698af7f (127.14.227.129:35625), 9db65ca094ab450daa6169167c78bc4c (127.14.227.130:45373)
I20260924 03:15:19.393998 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.373454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:35625 (local address 127.14.227.131:40131)
0924 03:15:19.384898 (+ 11444us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:19.384919 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:19.385037 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:19.386688 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:19.386721 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:19.386776 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:19.387022 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.387097 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.388865 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.388907 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.391288 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.391300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.393110 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.393116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.393287 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.393320 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:15:19.393399 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 03:15:19.393538 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11326,"thread_start_us":11684,"threads_started":1}
I20260924 03:15:19.394434 16880 raft_consensus.cc:482] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:19.395165 16880 raft_consensus.cc:504] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.395784 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.373741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:40131 (local address 127.14.227.129:35625)
0924 03:15:19.386330 (+ 12589us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.386349 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.386368 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:15:19.386431 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.386438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.386506 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.386607 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:19.387279 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.388640 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.391471 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.392763 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.392952 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.393025 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.394245 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.395426 (+ 1181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12447,"thread_start_us":12348,"threads_started":1}
I20260924 03:15:19.397030 16880 leader_election.cc:288] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9db65ca094ab450daa6169167c78bc4c (127.14.227.130:45373), 6d5c396071f745edb41c02b3598a9084 (127.14.227.131:35071)
I20260924 03:15:19.397930 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.374014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:32837 (local address 127.14.227.130:45373)
0924 03:15:19.374197 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.374202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.391254 (+ 17052us) server_negotiation.cc:407] Connection header received
0924 03:15:19.391531 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.391537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.391624 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.391688 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:19.392365 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.393446 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.395710 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.396678 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.396783 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.396831 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.397415 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.397572 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:15:19.399699 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.373688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45373 (local address 127.14.227.131:32837)
0924 03:15:19.391149 (+ 17461us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:19.391168 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:19.391361 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:19.391874 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:19.391879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:19.391903 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:19.392192 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.392204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.393608 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.393612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.395516 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.395532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.396990 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.396995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.397211 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.397264 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 03:15:19.399023 (+ 1759us) client_negotiation.cc:240] Negotiation successful
0924 03:15:19.399231 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12410,"thread_start_us":17235,"threads_started":1}
I20260924 03:15:19.400653 16669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343f859014a54138bc271b98905d23cc" candidate_uuid: "6d5c396071f745edb41c02b3598a9084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db65ca094ab450daa6169167c78bc4c" is_pre_election: true
I20260924 03:15:19.401093 16669 raft_consensus.cc:2454] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d5c396071f745edb41c02b3598a9084 in term 0.
I20260924 03:15:19.401814 16737 leader_election.cc:302] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5c396071f745edb41c02b3598a9084, 9db65ca094ab450daa6169167c78bc4c; no voters:
I20260924 03:15:19.402274 16881 raft_consensus.cc:2781] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:19.402405 16881 raft_consensus.cc:482] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:19.402496 16881 raft_consensus.cc:3037] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:19.405133 16881 raft_consensus.cc:504] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.405723 16881 leader_election.cc:288] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [CANDIDATE]: Term 1 election: Requested vote from peers dfa47051a69b4368835400db5698af7f (127.14.227.129:35625), 9db65ca094ab450daa6169167c78bc4c (127.14.227.130:45373)
I20260924 03:15:19.406325 16669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343f859014a54138bc271b98905d23cc" candidate_uuid: "6d5c396071f745edb41c02b3598a9084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db65ca094ab450daa6169167c78bc4c"
I20260924 03:15:19.406625 16669 raft_consensus.cc:3037] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:19.410405 16669 raft_consensus.cc:2454] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d5c396071f745edb41c02b3598a9084 in term 1.
I20260924 03:15:19.410789 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.398248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:53325 (local address 127.14.227.130:45373)
0924 03:15:19.398975 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.398983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.400878 (+ 1895us) server_negotiation.cc:407] Connection header received
0924 03:15:19.401083 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.401088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.401150 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.401252 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:19.405981 (+ 4729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.406844 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.408967 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.409796 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.410012 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.410091 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.410384 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.410522 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540}
I20260924 03:15:19.410837 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.397388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45373 (local address 127.14.227.129:53325)
0924 03:15:19.400765 (+ 3377us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:19.400788 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:19.400931 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:19.401609 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:19.401613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:19.401639 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:19.402037 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.402080 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.406997 (+ 4917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.407001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.408798 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.408829 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.410067 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.410074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.410224 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.410248 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:15:19.410347 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:15:19.410478 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3263,"thread_start_us":3120,"threads_started":1}
I20260924 03:15:19.410892 16737 leader_election.cc:302] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5c396071f745edb41c02b3598a9084, 9db65ca094ab450daa6169167c78bc4c; no voters:
I20260924 03:15:19.411343 16881 raft_consensus.cc:2781] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:19.411818 16881 raft_consensus.cc:686] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [term 1 LEADER]: Becoming Leader. State: Replica: 6d5c396071f745edb41c02b3598a9084, State: Running, Role: LEADER
I20260924 03:15:19.412246 16881 consensus_queue.cc:237] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } }
I20260924 03:15:19.414031 16669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343f859014a54138bc271b98905d23cc" candidate_uuid: "dfa47051a69b4368835400db5698af7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db65ca094ab450daa6169167c78bc4c" is_pre_election: true
I20260924 03:15:19.414417 16669 raft_consensus.cc:2379] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dfa47051a69b4368835400db5698af7f in current term 1: Already voted for candidate 6d5c396071f745edb41c02b3598a9084 in this term.
I20260924 03:15:19.416272 16920 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: TxnStatusTablet state changed. Reason: New leader 6d5c396071f745edb41c02b3598a9084. Latest consensus state: current_term: 1 leader_uuid: "6d5c396071f745edb41c02b3598a9084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.416891 16920 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:15:19.418238 16369 catalog_manager.cc:5495] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d5c396071f745edb41c02b3598a9084 (127.14.227.131). New cstate: current_term: 1 leader_uuid: "6d5c396071f745edb41c02b3598a9084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } health_report { overall_health: HEALTHY } } }
I20260924 03:15:19.419821 16530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343f859014a54138bc271b98905d23cc" candidate_uuid: "6d5c396071f745edb41c02b3598a9084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa47051a69b4368835400db5698af7f" is_pre_election: true
I20260924 03:15:19.420115 16530 raft_consensus.cc:2454] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d5c396071f745edb41c02b3598a9084 in term 0.
I20260924 03:15:19.420634 16530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343f859014a54138bc271b98905d23cc" candidate_uuid: "6d5c396071f745edb41c02b3598a9084" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa47051a69b4368835400db5698af7f"
I20260924 03:15:19.420856 16530 raft_consensus.cc:3037] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:19.423655 16530 raft_consensus.cc:2454] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d5c396071f745edb41c02b3598a9084 in term 1.
I20260924 03:15:19.426877 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.397836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:59541 (local address 127.14.227.131:35071)
0924 03:15:19.413078 (+ 15242us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.413084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.413789 (+ 705us) server_negotiation.cc:407] Connection header received
0924 03:15:19.414451 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.414476 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.414548 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.414615 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:19.415448 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.416845 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.424280 (+ 7435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.425636 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.425787 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.425854 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.426422 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.426549 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15052,"thread_start_us":15290,"threads_started":1}
I20260924 03:15:19.427457 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.398288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:35071 (local address 127.14.227.129:59541)
0924 03:15:19.413648 (+ 15360us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:19.413666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:19.414292 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:19.414886 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:19.414891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:19.414921 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:19.415271 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.415280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.417014 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.417030 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.424120 (+ 7090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.424134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.426065 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.426071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.426241 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.426272 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:15:19.427026 (+ 754us) client_negotiation.cc:240] Negotiation successful
0924 03:15:19.427160 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15244,"thread_start_us":21100,"threads_started":1}
I20260924 03:15:19.429438 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343f859014a54138bc271b98905d23cc" candidate_uuid: "dfa47051a69b4368835400db5698af7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5c396071f745edb41c02b3598a9084" is_pre_election: true
I20260924 03:15:19.429654 16922 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:15:19.429876 16922 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:15:19.430228 16459 leader_election.cc:302] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa47051a69b4368835400db5698af7f; no voters: 6d5c396071f745edb41c02b3598a9084, 9db65ca094ab450daa6169167c78bc4c
I20260924 03:15:19.431316 16880 raft_consensus.cc:2726] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 03:15:19.440323 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.432921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.14.227.131:35071)
0924 03:15:19.433557 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.433563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.433579 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:15:19.433671 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.433676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.433767 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.433899 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:19.434358 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.435731 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.437660 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.438343 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.438472 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.439022 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:19.439709 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:19.439868 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.440030 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.440111 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526}
I20260924 03:15:19.440440 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.432808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:35071 (local address 127.0.0.1:45772)
0924 03:15:19.433009 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:19.433036 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:19.433148 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:19.433943 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:19.433983 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:19.433998 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:19.434194 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.434207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.435905 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.435912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.437506 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.437524 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.438494 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.438501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.438752 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.438820 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:19.439850 (+ 1030us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:19.439871 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 03:15:19.439993 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:15:19.440154 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 03:15:19.444248 16530 raft_consensus.cc:1261] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Refusing update from remote peer 6d5c396071f745edb41c02b3598a9084: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:19.444315 16669 raft_consensus.cc:1261] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Refusing update from remote peer 6d5c396071f745edb41c02b3598a9084: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:19.444912 16920 consensus_queue.cc:1035] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:19.445271 16881 consensus_queue.cc:1035] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:19.452975 16893 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: TxnStatusTablet state changed. Reason: New leader 6d5c396071f745edb41c02b3598a9084. Latest consensus state: current_term: 1 leader_uuid: "6d5c396071f745edb41c02b3598a9084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.453282 16893 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:19.454779 16925 mvcc.cc:205] Tried to move back new op lower bound from 7332739970837327872 to 7332739970717417472. Current Snapshot: MvccSnapshot[applied={T|T < 7332739970837327872}]
I20260924 03:15:19.456243 16926 mvcc.cc:205] Tried to move back new op lower bound from 7332739970837327872 to 7332739970717417472. Current Snapshot: MvccSnapshot[applied={T|T < 7332739970837327872}]
I20260924 03:15:19.456070 16880 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: TxnStatusTablet state changed. Reason: New leader 6d5c396071f745edb41c02b3598a9084. Latest consensus state: current_term: 1 leader_uuid: "6d5c396071f745edb41c02b3598a9084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.456384 16880 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:19.457417 16893 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d5c396071f745edb41c02b3598a9084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.457695 16893 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P 9db65ca094ab450daa6169167c78bc4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:19.461314 16920 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d5c396071f745edb41c02b3598a9084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.461603 16920 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:15:19.461819 16920 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d5c396071f745edb41c02b3598a9084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.462065 16920 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P 6d5c396071f745edb41c02b3598a9084: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:15:19.471518 16880 tablet_replica.cc:439] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d5c396071f745edb41c02b3598a9084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 } } peers { permanent_uuid: "9db65ca094ab450daa6169167c78bc4c" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45373 } } peers { permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 } } }
I20260924 03:15:19.471807 16880 tablet_replica.cc:442] T 343f859014a54138bc271b98905d23cc P dfa47051a69b4368835400db5698af7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:19.488351 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.482311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:41413 (local address 127.14.227.131:35071)
0924 03:15:19.482689 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.482694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.482894 (+ 200us) server_negotiation.cc:407] Connection header received
0924 03:15:19.483079 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.483082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.483158 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.483227 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:19.483780 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.484536 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.486563 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.486966 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.487099 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.487680 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:19.487803 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:19.487872 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.488058 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.488126 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260924 03:15:19.488581 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.482198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:35071 (local address 127.14.227.130:41413)
0924 03:15:19.482793 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:19.482814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:19.482961 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:19.483274 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:19.483279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:19.483294 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:19.483619 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.483644 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.484660 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.484665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.486430 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.486446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.487129 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.487135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.487370 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.487482 (+ 112us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:19.487903 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:19.487915 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:15:19.488026 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:15:19.488176 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":252,"threads_started":1}
I20260924 03:15:19.498034 16809 raft_consensus.cc:1261] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Refusing update from remote peer 9db65ca094ab450daa6169167c78bc4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:19.498286 16530 raft_consensus.cc:1261] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Refusing update from remote peer 9db65ca094ab450daa6169167c78bc4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:19.498615 16893 consensus_queue.cc:1035] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5c396071f745edb41c02b3598a9084" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:19.498855 16884 consensus_queue.cc:1035] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa47051a69b4368835400db5698af7f" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:19.532436 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.525193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:39187 (local address 127.14.227.130:45373)
0924 03:15:19.525367 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.525375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.526133 (+ 758us) server_negotiation.cc:407] Connection header received
0924 03:15:19.526294 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.526297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.526353 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.526422 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:19.527071 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.528138 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.530434 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.531077 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.531174 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.531818 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:19.531911 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:19.531979 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.532131 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.532189 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:15:19.532831 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.525316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45373 (local address 127.14.227.131:39187)
0924 03:15:19.526037 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:19.526057 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:19.526194 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:19.526456 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:19.526461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:19.526474 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:19.526897 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.526909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.528497 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.528502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.530258 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:19.530278 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.531354 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.531360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:19.531575 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.531639 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:19.532006 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:19.532017 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:15:19.532121 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:15:19.532264 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":339,"threads_started":1}
I20260924 03:15:19.573176 16369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:19.573436 16369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:19.576323 16369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1d7f7f6f48444b8a14c1ef265750b2c: Sending DeleteTablet for 3 replicas of tablet c4712627d7874279ae880eaf8c861862
I20260924 03:15:19.577090 16644 tablet_service.cc:1514] Processing DeleteTablet for tablet c4712627d7874279ae880eaf8c861862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:19 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:46004
I20260924 03:15:19.577174 16506 tablet_service.cc:1514] Processing DeleteTablet for tablet c4712627d7874279ae880eaf8c861862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:19 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38854
I20260924 03:15:19.577478 16783 tablet_service.cc:1514] Processing DeleteTablet for tablet c4712627d7874279ae880eaf8c861862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:19 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:45758
I20260924 03:15:19.588176 16946 tablet_replica.cc:330] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: stopping tablet replica
I20260924 03:15:19.588464 16946 raft_consensus.cc:2229] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:19.588815 16946 raft_consensus.cc:2258] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:19.589733 16944 tablet_replica.cc:330] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: stopping tablet replica
I20260924 03:15:19.590003 16944 raft_consensus.cc:2229] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:15:19.590314 16946 ts_tablet_manager.cc:1905] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:19.590492 16944 raft_consensus.cc:2258] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:19.591565 16944 ts_tablet_manager.cc:1905] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:19.597617 16946 ts_tablet_manager.cc:1918] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:19.597752 16946 log.cc:1196] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/c4712627d7874279ae880eaf8c861862
I20260924 03:15:19.598285 16946 ts_tablet_manager.cc:1939] T c4712627d7874279ae880eaf8c861862 P 6d5c396071f745edb41c02b3598a9084: Deleting consensus metadata
I20260924 03:15:19.598526 16944 ts_tablet_manager.cc:1918] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:19.598724 16944 log.cc:1196] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/c4712627d7874279ae880eaf8c861862
I20260924 03:15:19.598987 16945 tablet_replica.cc:330] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: stopping tablet replica
I20260924 03:15:19.599218 16945 raft_consensus.cc:2229] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:19.599345 16944 ts_tablet_manager.cc:1939] T c4712627d7874279ae880eaf8c861862 P 9db65ca094ab450daa6169167c78bc4c: Deleting consensus metadata
I20260924 03:15:19.599462 16945 raft_consensus.cc:2258] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:19.600467 16355 catalog_manager.cc:4841] TS 9db65ca094ab450daa6169167c78bc4c (127.14.227.130:45373): tablet c4712627d7874279ae880eaf8c861862 (table test-table [id=d5dd2e3b10eb49819038741ee2012c95]) successfully deleted
I20260924 03:15:19.600567 16945 ts_tablet_manager.cc:1905] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:19.606300 16945 ts_tablet_manager.cc:1918] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:19.606446 16945 log.cc:1196] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/c4712627d7874279ae880eaf8c861862
I20260924 03:15:19.606899 16945 ts_tablet_manager.cc:1939] T c4712627d7874279ae880eaf8c861862 P dfa47051a69b4368835400db5698af7f: Deleting consensus metadata
I20260924 03:15:19.607368 16356 catalog_manager.cc:4841] TS 6d5c396071f745edb41c02b3598a9084 (127.14.227.131:35071): tablet c4712627d7874279ae880eaf8c861862 (table test-table [id=d5dd2e3b10eb49819038741ee2012c95]) successfully deleted
I20260924 03:15:19.608002 16356 catalog_manager.cc:4841] TS dfa47051a69b4368835400db5698af7f (127.14.227.129:35625): tablet c4712627d7874279ae880eaf8c861862 (table test-table [id=d5dd2e3b10eb49819038741ee2012c95]) successfully deleted
Sep 24 03:15:19 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:19.617904 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.599948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.14.227.129:35625)
0924 03:15:19.600119 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.600126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.601453 (+ 1327us) server_negotiation.cc:407] Connection header received
0924 03:15:19.601637 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.601646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.601779 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.601912 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:19.603954 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.605099 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.607009 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.607715 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.607860 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.612317 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:19.612328 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:19.612343 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:19.612374 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:19.615199 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.615998 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.616005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.616007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.616119 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.616515 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.616518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.616520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.616841 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:19.617115 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.617418 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.617579 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 03:15:19.618693 16571 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38866
I20260924 03:15:19.635689 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.621410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.14.227.190:34525)
0924 03:15:19.621639 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.621644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.621798 (+ 154us) server_negotiation.cc:407] Connection header received
0924 03:15:19.622009 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.622014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.622107 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.622216 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:19.623467 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.624645 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.625775 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.626487 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.626816 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.628986 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:19.628993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:19.628998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:19.629040 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:19.632753 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.633503 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.633509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.633514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.633729 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.634106 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.634112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.634116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.634528 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:19.634793 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.635044 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.635298 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 03:15:19.636312 16369 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45902
I20260924 03:15:19.638831 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16440
I20260924 03:15:19.653728 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.639770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.14.227.129:35625)
0924 03:15:19.639975 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.639982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.640236 (+ 254us) server_negotiation.cc:407] Connection header received
0924 03:15:19.640406 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.640412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.640510 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.640637 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:19.641751 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.643062 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.644185 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.644908 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.645047 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.647332 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:19.647340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:19.647346 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:19.647388 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:19.651038 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.651647 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.651653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.651658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.651809 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.652122 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.652128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.652132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.652516 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:19.652777 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.653064 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.653300 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 03:15:19.654381 16571 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38874
W20260924 03:15:19.654929 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16440: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:19.655139 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 16440
I20260924 03:15:19.672351 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16580
Sep 24 03:15:19 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:15:19.689826 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.673178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.14.227.130:45373)
0924 03:15:19.673335 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.673339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.673356 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:15:19.673562 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.673566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.673626 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.673716 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:19.675390 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.676124 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.678107 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.678508 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.678597 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.684065 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:19.684078 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:19.684091 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:19.684128 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:19.686866 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.687719 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.687725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.687727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.687882 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.688298 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.688301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.688303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.688609 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:19.688975 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.689276 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.689493 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 03:15:19.690685 16710 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46014
W20260924 03:15:19.691386 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16580: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:19.691557 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 16580
I20260924 03:15:19.714365 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16719
Sep 24 03:15:19 dist-test-slave-vd0p krb5kdc[16324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:15:19.734627 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:19.715288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.14.227.131:35071)
0924 03:15:19.715502 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:19.715515 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:19.715538 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:15:19.715744 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:19.715752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:19.715844 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:19.715958 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:19.717543 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.718999 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.720795 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:19.721504 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:19.721637 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:19.727144 (+ 5507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:19.727162 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:19.727180 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:19.727227 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:19.731286 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.732168 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.732176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.732180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.732334 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:19.732766 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:19.732771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:19.732774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:19.733300 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:19.733587 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:19.733905 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 03:15:19.734133 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 03:15:19.735375 16849 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778
W20260924 03:15:19.736130 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16719: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:19.736303 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 16719
I20260924 03:15:19.755008 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16337
I20260924 03:15:19.756136 16409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:19.868741 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 16337
2026-09-24T03:15:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4433 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:19 dist-test-slave-vd0p krb5kdc[16955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:19 dist-test-slave-vd0p krb5kdc[16955](info): set up 2 sockets
Sep 24 03:15:19 dist-test-slave-vd0p krb5kdc[16955](info): commencing operation
krb5kdc: starting...
W20260924 03:15:20.946115 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:21 dist-test-slave-vd0p krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:21.124933 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44129
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:44129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:21.437160 16968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:21.437489 16968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:21.437650 16968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:21.445099 16968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:21.445261 16968 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:15:21.445359 16968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:21.445475 16968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:21.445564 16968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:21.445652 16968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:21.456491 16968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:44129
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:21.457784 16968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:21.459244 16968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:21.470603 16973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:21.470561 16976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:21.470630 16974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:21.471401 16968 server_base.cc:1034] running on GCE node
I20260924 03:15:21.472196 16968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:21.473595 16968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:21.474831 16968 hybrid_clock.cc:648] HybridClock initialized: now 1790219721474734 us; error 71 us; skew 500 ppm
I20260924 03:15:21.475178 16968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:21 dist-test-slave-vd0p krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219721, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:21.478179 16968 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:21.479666 16968 webserver.cc:460] Webserver started at http://127.14.227.190:36313/ using document root <none> and password file <none>
I20260924 03:15:21.480356 16968 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:21.480471 16968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:21.480777 16968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:21.483664 16968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4"
format_stamp: "Formatted at 2026-09-24 03:15:21 on dist-test-slave-vd0p"
server_key: "f8ace86eb32a5f730542807f225bb2c5"
server_key_iv: "feec44658141442da68b702c2a685a51"
server_key_version: "encryptionkey@0"
I20260924 03:15:21.484408 16968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4"
format_stamp: "Formatted at 2026-09-24 03:15:21 on dist-test-slave-vd0p"
server_key: "f8ace86eb32a5f730542807f225bb2c5"
server_key_iv: "feec44658141442da68b702c2a685a51"
server_key_version: "encryptionkey@0"
I20260924 03:15:21.489367 16968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:15:21.492800 16983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:21.494138 16968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:21.494297 16968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4"
format_stamp: "Formatted at 2026-09-24 03:15:21 on dist-test-slave-vd0p"
server_key: "f8ace86eb32a5f730542807f225bb2c5"
server_key_iv: "feec44658141442da68b702c2a685a51"
server_key_version: "encryptionkey@0"
I20260924 03:15:21.494505 16968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:21.522811 16968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:21.526731 16968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:21.527004 16968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:21.548014 16968 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:44129
I20260924 03:15:21.548030 17044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:44129 every 8 connection(s)
I20260924 03:15:21.549531 16968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:21.550217 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 16968
I20260924 03:15:21.550446 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:21.551213 15246 external_mini_cluster.cc:1428] Setting key d286c244990075592f68aa55087198ef
I20260924 03:15:21.554016 17045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:15:21 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:21.567358 17045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: Bootstrap starting.
I20260924 03:15:21.571427 17045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:21.572742 17045 log.cc:824] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:21.576187 17045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: No bootstrap required, opened a new log
I20260924 03:15:21.577024 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:21.553288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.14.227.190:44129)
0924 03:15:21.554097 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:21.554132 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:21.554181 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:15:21.556447 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:21.556476 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:21.557000 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:21.557451 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:21.559321 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:21.561397 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:21.562815 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:21.563894 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:21.564066 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:21.568356 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:21.568366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:21.568384 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:21.568433 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:21.572382 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:21.573108 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:21.573115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:21.573123 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:21.573349 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:21.573832 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:21.573835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:21.573837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:21.574544 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:21.574823 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:21.575206 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 03:15:21.575635 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":310,"threads_started":1}
I20260924 03:15:21.584249 17045 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44129 } }
I20260924 03:15:21.584611 17045 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:21.584690 17045 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e1f0d6e0314ee4934f77ccb4b1fcd4, State: Initialized, Role: FOLLOWER
I20260924 03:15:21.585248 17045 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44129 } }
I20260924 03:15:21.585457 17045 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:21.585551 17045 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:21.585666 17045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:21.587975 17045 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44129 } }
I20260924 03:15:21.588403 17045 leader_election.cc:302] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7e1f0d6e0314ee4934f77ccb4b1fcd4; no voters:
I20260924 03:15:21.588907 17045 leader_election.cc:288] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:21.589059 17050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:21.589722 17050 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [term 1 LEADER]: Becoming Leader. State: Replica: a7e1f0d6e0314ee4934f77ccb4b1fcd4, State: Running, Role: LEADER
I20260924 03:15:21.590226 17050 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44129 } }
I20260924 03:15:21.590902 17045 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:21.592881 17052 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e1f0d6e0314ee4934f77ccb4b1fcd4. Latest consensus state: current_term: 1 leader_uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44129 } } }
I20260924 03:15:21.593271 17052 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:21.594825 17051 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e1f0d6e0314ee4934f77ccb4b1fcd4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44129 } } }
I20260924 03:15:21.595440 17051 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:21.596199 17058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:21.601263 17058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:21.609699 17058 catalog_manager.cc:1347] Generated new cluster ID: 4d298edbb5d8460e9c30fe903a5f292f
I20260924 03:15:21.609817 17058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:21.636658 17058 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:21.638774 17058 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:21.660153 17058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: Generated new TSK 0
I20260924 03:15:21.660969 17058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:21.719884 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:22.067549 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:22.067803 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:22.067903 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:22.074924 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:22.075024 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:22.075124 17071 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:15:22.075176 17071 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:15:22.075249 17071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:22.084695 17071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:44129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:22.085886 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:22.087180 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:22.100064 17079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:22.100025 17077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:22.100106 17076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:22.101121 17071 server_base.cc:1034] running on GCE node
I20260924 03:15:22.101655 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:22.102628 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:22.103817 17071 hybrid_clock.cc:648] HybridClock initialized: now 1790219722103783 us; error 34 us; skew 500 ppm
I20260924 03:15:22.104115 17071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:22 dist-test-slave-vd0p krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219722, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:22.107120 17071 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:22.108462 17071 webserver.cc:460] Webserver started at http://127.14.227.129:36331/ using document root <none> and password file <none>
I20260924 03:15:22.109103 17071 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:22.109241 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:22.109537 17071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:22.112351 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "6e233f82279040c382b829f3ec3edc92"
format_stamp: "Formatted at 2026-09-24 03:15:22 on dist-test-slave-vd0p"
server_key: "8793e60b133cbff429813a388eb0f4c4"
server_key_iv: "36b0b40052447f4d7b9f9d4a520cd2a8"
server_key_version: "encryptionkey@0"
I20260924 03:15:22.113063 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "6e233f82279040c382b829f3ec3edc92"
format_stamp: "Formatted at 2026-09-24 03:15:22 on dist-test-slave-vd0p"
server_key: "8793e60b133cbff429813a388eb0f4c4"
server_key_iv: "36b0b40052447f4d7b9f9d4a520cd2a8"
server_key_version: "encryptionkey@0"
I20260924 03:15:22.117856 17071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 03:15:22.121073 17086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:22.122298 17071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 03:15:22.122458 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "6e233f82279040c382b829f3ec3edc92"
format_stamp: "Formatted at 2026-09-24 03:15:22 on dist-test-slave-vd0p"
server_key: "8793e60b133cbff429813a388eb0f4c4"
server_key_iv: "36b0b40052447f4d7b9f9d4a520cd2a8"
server_key_version: "encryptionkey@0"
I20260924 03:15:22.122654 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:22.151319 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:22.155225 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:22.155509 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:22.157867 17071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:22.158025 17071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:22.158159 17071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:22.158263 17071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:15:22 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219722, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:22.180091 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:22.158686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:38813 (local address 127.14.227.190:44129)
0924 03:15:22.159436 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:22.159444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:22.160476 (+ 1032us) server_negotiation.cc:407] Connection header received
0924 03:15:22.163518 (+ 3042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:22.163525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:22.163611 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:22.163749 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:22.165297 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.166248 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.169443 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.169994 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.170137 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:22.174091 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:22.174098 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:22.174103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:22.174145 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:22.177100 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:22.177837 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:22.177845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:22.177849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:22.178038 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:22.178440 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:22.178448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:22.178453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:22.178727 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:22.178868 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:22.179546 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 03:15:22.179750 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":518,"threads_started":1}
I20260924 03:15:22.181027 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:22.159348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44129 (local address 127.14.227.129:38813)
0924 03:15:22.160291 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:22.160348 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:22.163151 (+ 2803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:22.163987 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:22.163995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:22.164540 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:22.165131 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:22.165147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.166471 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.166481 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:22.169269 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:22.169293 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.170269 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.170278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:22.170627 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:22.171443 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:22.171467 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:22.173866 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:22.177235 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:22.177244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:22.177258 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:22.177718 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:22.178150 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:22.178155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:22.178159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:22.178336 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:22.178854 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:22.178861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:22.179221 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 03:15:22.179472 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 03:15:22.179959 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":393,"threads_started":1}
I20260924 03:15:22.199923 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:22.187198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44129 (local address 127.14.227.129:40665)
0924 03:15:22.187815 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:22.187831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:22.187976 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:22.188314 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:22.188319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:22.188755 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:22.189044 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:22.189053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.190089 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.190093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:22.191112 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:22.191122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.191913 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.191919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:22.192094 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:22.192688 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:22.192704 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:22.193358 (+ 654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:22.196747 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:22.196752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:22.196756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:22.197394 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:22.197882 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:22.197890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:22.197896 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:22.198072 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:22.198626 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:22.198631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:22.198853 (+ 222us) client_negotiation.cc:769] Sending connection context
0924 03:15:22.199067 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 03:15:22.199402 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":352,"threads_started":1}
I20260924 03:15:22.200310 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:22.187075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:40665 (local address 127.14.227.190:44129)
0924 03:15:22.187212 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:22.187216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:22.187910 (+ 694us) server_negotiation.cc:407] Connection header received
0924 03:15:22.188116 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:22.188121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:22.188191 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:22.188284 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:22.189173 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.189929 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.191236 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.191721 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.191873 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:22.193530 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:22.193536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:22.193541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:22.193579 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:22.196627 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:22.197540 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:22.197547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:22.197553 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:22.197725 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:22.198199 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:22.198206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:22.198212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:22.198512 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:22.198662 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:22.199726 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0924 03:15:22.199958 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:15:22.216964 17071 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:39783
I20260924 03:15:22.217001 17206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:39783 every 8 connection(s)
I20260924 03:15:22.218218 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:22.219993 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 17071
I20260924 03:15:22.220209 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:22.220247 17207 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44129
I20260924 03:15:22.220472 17207 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:22.220976 15246 external_mini_cluster.cc:1428] Setting key adb9cc21391695de03ab1012a49adeee
I20260924 03:15:22.221155 17207 heartbeater.cc:511] Master 127.14.227.190:44129 requested a full tablet report, sending...
I20260924 03:15:22.223407 16999 ts_manager.cc:194] Registered new tserver with Master: 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783)
I20260924 03:15:22.225651 16999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:38813
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:22.260375 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:22.569314 17211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:22.569554 17211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:22.569730 17211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:22.576741 17211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:22.576853 17211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:22.577003 17211 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:15:22.577078 17211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:15:22.577200 17211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:22.586779 17211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:44129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:22.587962 17211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:22.589221 17211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:22.601928 17216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:22.601836 17219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:22.601873 17217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:22.602670 17211 server_base.cc:1034] running on GCE node
I20260924 03:15:22.603281 17211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:22.604177 17211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:22.605370 17211 hybrid_clock.cc:648] HybridClock initialized: now 1790219722605329 us; error 30 us; skew 500 ppm
I20260924 03:15:22.605684 17211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:22 dist-test-slave-vd0p krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219722, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:22.608702 17211 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:22.610070 17211 webserver.cc:460] Webserver started at http://127.14.227.130:45017/ using document root <none> and password file <none>
I20260924 03:15:22.610731 17211 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:22.610853 17211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:22.611156 17211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:22.613912 17211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "b17cb4c30ad8455e89135d44a5ea6c0b"
format_stamp: "Formatted at 2026-09-24 03:15:22 on dist-test-slave-vd0p"
server_key: "bdf2ab1ef125e96bdd5ae934a53445b2"
server_key_iv: "d40463e14cdfbf7d8653a57e38bfd2ef"
server_key_version: "encryptionkey@0"
I20260924 03:15:22.614689 17211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "b17cb4c30ad8455e89135d44a5ea6c0b"
format_stamp: "Formatted at 2026-09-24 03:15:22 on dist-test-slave-vd0p"
server_key: "bdf2ab1ef125e96bdd5ae934a53445b2"
server_key_iv: "d40463e14cdfbf7d8653a57e38bfd2ef"
server_key_version: "encryptionkey@0"
I20260924 03:15:22.619618 17211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:15:22.622902 17226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:22.624143 17211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:15:22.624307 17211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "b17cb4c30ad8455e89135d44a5ea6c0b"
format_stamp: "Formatted at 2026-09-24 03:15:22 on dist-test-slave-vd0p"
server_key: "bdf2ab1ef125e96bdd5ae934a53445b2"
server_key_iv: "d40463e14cdfbf7d8653a57e38bfd2ef"
server_key_version: "encryptionkey@0"
I20260924 03:15:22.624490 17211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:22.639614 17211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:22.643461 17211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:22.643746 17211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:22.646301 17211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:22.646435 17211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:22.646555 17211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:22.646662 17211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:15:22 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219722, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:22.670697 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:22.647210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:39409 (local address 127.14.227.190:44129)
0924 03:15:22.647441 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:22.647449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:22.649056 (+ 1607us) server_negotiation.cc:407] Connection header received
0924 03:15:22.651562 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:22.651568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:22.651661 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:22.651767 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:22.654207 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.655424 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.657557 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.658444 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.658603 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:22.663533 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:22.663545 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:22.663552 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:22.663594 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:22.667349 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:22.668034 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:22.668042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:22.668048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:22.668200 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:22.668774 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:22.668780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:22.668783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:22.669200 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:22.669374 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:22.670120 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 03:15:22.670349 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 03:15:22.672222 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:22.647781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44129 (local address 127.14.227.130:39409)
0924 03:15:22.648870 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:22.648925 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:22.651204 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:22.652037 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:22.652052 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:22.653103 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:22.654050 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:22.654069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.655629 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.655636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:22.657371 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:22.657405 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.658694 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.658704 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:22.658978 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:22.659750 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:22.659774 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:22.663268 (+ 3494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:22.667486 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:22.667511 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:22.667524 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:22.667908 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:22.668321 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:22.668327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:22.668332 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:22.668655 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:22.669336 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:22.669345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:22.669693 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 03:15:22.669985 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 03:15:22.670561 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":617,"threads_started":1}
I20260924 03:15:22.692154 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:22.678201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44129 (local address 127.14.227.130:41427)
0924 03:15:22.678771 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:22.678788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:22.678945 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:22.679239 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:22.679243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:22.679682 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:22.679966 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:22.679975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.680927 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.680933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:22.682153 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:22.682165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.683139 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.683146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:22.683320 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:22.684163 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:22.684181 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:22.685441 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:22.688836 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:22.688841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:22.688848 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:22.689343 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:22.689788 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:22.689795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:22.689799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:22.689940 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:22.690740 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:22.690745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:22.690962 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 03:15:22.691213 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 03:15:22.691554 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":329,"threads_started":1}
I20260924 03:15:22.692727 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:22.678068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:41427 (local address 127.14.227.190:44129)
0924 03:15:22.678212 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:22.678216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:22.678871 (+ 655us) server_negotiation.cc:407] Connection header received
0924 03:15:22.679062 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:22.679071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:22.679126 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:22.679218 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:22.680094 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.680794 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.682287 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:22.682915 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:22.683093 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:22.685675 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:22.685679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:22.685682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:22.685712 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:22.688712 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:22.689468 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:22.689476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:22.689482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:22.689666 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:22.690081 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:22.690087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:22.690090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:22.690535 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:22.690679 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:22.692261 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0924 03:15:22.692481 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:15:22.696853 17211 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:45953
I20260924 03:15:22.696931 17345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:45953 every 8 connection(s)
I20260924 03:15:22.698189 17211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:22.700294 17346 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44129
I20260924 03:15:22.700513 17346 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:22.701150 17346 heartbeater.cc:511] Master 127.14.227.190:44129 requested a full tablet report, sending...
I20260924 03:15:22.702411 16999 ts_manager.cc:194] Registered new tserver with Master: b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953)
I20260924 03:15:22.703209 16999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:39409
I20260924 03:15:22.706735 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 17211
I20260924 03:15:22.706876 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:22.707257 15246 external_mini_cluster.cc:1428] Setting key 97d88134db0fc341f770c31e8f1e6f98
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:22.745316 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:23.065299 17350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:23.065554 17350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:23.065706 17350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:23.073225 17350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:23.073347 17350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:23.073491 17350 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 03:15:23.073565 17350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:15:23.073693 17350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:23.084307 17350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:44129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:23.085561 17350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:23.086889 17350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:23.099960 17355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:23.099917 17358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:23.100018 17356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:23.101168 17350 server_base.cc:1034] running on GCE node
I20260924 03:15:23.101764 17350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:23.102665 17350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:23.103850 17350 hybrid_clock.cc:648] HybridClock initialized: now 1790219723103811 us; error 32 us; skew 500 ppm
I20260924 03:15:23.104164 17350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:23 dist-test-slave-vd0p krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219723, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:23.107199 17350 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:23.108529 17350 webserver.cc:460] Webserver started at http://127.14.227.131:44623/ using document root <none> and password file <none>
I20260924 03:15:23.109184 17350 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:23.109311 17350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:23.109632 17350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:23.112452 17350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "450df1629adc458ba9d3cd8f3f6008fb"
format_stamp: "Formatted at 2026-09-24 03:15:23 on dist-test-slave-vd0p"
server_key: "f93a41a9817c7568a2d9ec30c6f5e566"
server_key_iv: "b42f189bef95cdf3eb1c6265432f04f9"
server_key_version: "encryptionkey@0"
I20260924 03:15:23.113165 17350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "450df1629adc458ba9d3cd8f3f6008fb"
format_stamp: "Formatted at 2026-09-24 03:15:23 on dist-test-slave-vd0p"
server_key: "f93a41a9817c7568a2d9ec30c6f5e566"
server_key_iv: "b42f189bef95cdf3eb1c6265432f04f9"
server_key_version: "encryptionkey@0"
I20260924 03:15:23.118131 17350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:15:23.121351 17365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:23.122749 17350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:23.122915 17350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "450df1629adc458ba9d3cd8f3f6008fb"
format_stamp: "Formatted at 2026-09-24 03:15:23 on dist-test-slave-vd0p"
server_key: "f93a41a9817c7568a2d9ec30c6f5e566"
server_key_iv: "b42f189bef95cdf3eb1c6265432f04f9"
server_key_version: "encryptionkey@0"
I20260924 03:15:23.123107 17350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:23.150177 17350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:23.154066 17350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:23.154349 17350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:23.156927 17350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:23.157021 17350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:23.157166 17350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:23.157296 17350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 03:15:23 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219723, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:23.182107 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.157353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:40743 (local address 127.14.227.190:44129)
0924 03:15:23.157571 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.157581 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.159143 (+ 1562us) server_negotiation.cc:407] Connection header received
0924 03:15:23.161268 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.161277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.161388 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.161520 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:23.164423 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.165390 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.168689 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.169205 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.169347 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.174455 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:23.174465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:23.174471 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:23.174543 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:23.177896 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:23.179455 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:23.179462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:23.179465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:23.179622 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:23.180257 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:23.180263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:23.180266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:23.180549 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:23.180715 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.181459 (+ 744us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.181681 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:15:23.183226 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.158130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44129 (local address 127.14.227.131:40743)
0924 03:15:23.158971 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.159015 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.160907 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.162071 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.162085 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.163129 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:23.164181 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.164216 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.165585 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.165607 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.168496 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.168516 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.169589 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.169594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.169949 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.170818 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:23.170867 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:23.174179 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:23.178188 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:23.178205 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:23.178230 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:23.179271 (+ 1041us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:23.179785 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:23.179793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:23.179803 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:23.180122 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:23.180680 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:23.180692 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:23.181080 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.181350 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.181931 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":360,"threads_started":1}
I20260924 03:15:23.203754 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.188947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:48795 (local address 127.14.227.190:44129)
0924 03:15:23.189520 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.189528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.190068 (+ 540us) server_negotiation.cc:407] Connection header received
0924 03:15:23.190303 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.190310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.190390 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.190498 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:23.191579 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.192374 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.193775 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.194285 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.194445 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.197598 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:23.197603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:23.197606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:23.197658 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:23.200446 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:23.201409 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:23.201412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:23.201414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:23.201529 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:23.201997 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:23.202004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:23.202008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:23.202380 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:23.202535 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.203114 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.203387 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260924 03:15:23.203850 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.189139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44129 (local address 127.14.227.131:48795)
0924 03:15:23.189916 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.189972 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.190154 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.190529 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.190534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.191129 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:23.191444 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.191453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.192510 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.192517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.193632 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.193644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.194523 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.194535 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.194862 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.195711 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:23.195744 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:23.197283 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:23.200576 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:23.200592 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:23.200599 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:23.201287 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:23.201644 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:23.201651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:23.201657 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:23.201847 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:23.202497 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:23.202502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:23.202757 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.202949 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.203378 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":409,"threads_started":1}
I20260924 03:15:23.214897 17350 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:41433
I20260924 03:15:23.214933 17484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:41433 every 8 connection(s)
I20260924 03:15:23.216130 17350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:23.218190 17485 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44129
I20260924 03:15:23.218422 17485 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:23.219033 17485 heartbeater.cc:511] Master 127.14.227.190:44129 requested a full tablet report, sending...
I20260924 03:15:23.220323 16999 ts_manager.cc:194] Registered new tserver with Master: 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433)
I20260924 03:15:23.221381 16999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:40743
I20260924 03:15:23.222718 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 17350
I20260924 03:15:23.222882 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:23.223325 15246 external_mini_cluster.cc:1428] Setting key d3106b83ab565f4288f3c61aecdfcf4c
I20260924 03:15:23.226028 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:23.228641 17207 heartbeater.cc:503] Master 127.14.227.190:44129 was elected leader, sending a full tablet report...
Sep 24 03:15:23 dist-test-slave-vd0p krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:15:23 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:23.271487 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.258217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.14.227.190:44129)
0924 03:15:23.258384 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.258388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.258759 (+ 371us) server_negotiation.cc:407] Connection header received
0924 03:15:23.258914 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.258919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.258980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.259072 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:23.260394 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.261141 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.262372 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.262849 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.262986 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.266777 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:23.266782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:23.266785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:23.266815 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:23.269395 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:23.269938 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:23.269942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:23.269985 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:23.270102 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:23.270423 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:23.270426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:23.270428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:23.270696 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:23.270870 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.271073 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.271223 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:15:23.275610 16999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:23.283806 16999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:23.315861 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.296195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.14.227.130:45953)
0924 03:15:23.306218 (+ 10023us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.306230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.306270 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:15:23.306389 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.306397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.306723 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.308678 (+ 1955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.308918 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.310548 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.313190 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.314396 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.314592 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.314776 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.315372 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.315565 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9839,"thread_start_us":9714,"threads_started":1}
I20260924 03:15:23.315900 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.295689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45953 (local address 127.0.0.1:57734)
0924 03:15:23.302577 (+ 6888us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.302620 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.302781 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.307798 (+ 5017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.307806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.307838 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.308447 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.308461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.310699 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.310706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.313046 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.313062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.314920 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.314926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.315116 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.315168 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.315302 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.315483 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6646,"mutex_wait_us":1,"thread_start_us":6480,"threads_started":1}
I20260924 03:15:23.317728 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.303261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:41433 (local address 127.0.0.1:38604)
0924 03:15:23.309185 (+ 5924us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.309205 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.309328 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.309863 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.309869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.309896 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.310796 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.310814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.312422 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.312428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.314877 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.314890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.316751 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.316758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.316962 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.317097 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.317198 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.317330 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5803,"mutex_wait_us":59,"thread_start_us":5638,"threads_started":1}
I20260924 03:15:23.317727 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.296578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.14.227.129:39783)
0924 03:15:23.297214 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.297227 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.306640 (+ 9413us) server_negotiation.cc:407] Connection header received
0924 03:15:23.306964 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.306972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.307230 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.307409 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.310454 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.312004 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.314911 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.316327 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.316515 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.316671 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.317289 (+ 618us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.317457 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":248,"threads_started":1}
I20260924 03:15:23.317790 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.303372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.14.227.131:41433)
0924 03:15:23.308735 (+ 5363us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.308744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.309305 (+ 561us) server_negotiation.cc:407] Connection header received
0924 03:15:23.309502 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.309508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.309675 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.309902 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.311069 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.312270 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.315057 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.316520 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.316756 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.316923 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.317257 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.317419 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5171,"thread_start_us":4981,"threads_started":1}
I20260924 03:15:23.318825 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.295948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39783 (local address 127.0.0.1:56398)
0924 03:15:23.306510 (+ 10562us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.306529 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.306748 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.307943 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.307950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.307982 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.309801 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.309816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.312197 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.312202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.314745 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.314761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.316814 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.316820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.317088 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.317122 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.318384 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.318502 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10441,"mutex_wait_us":1420,"thread_start_us":10933,"threads_started":1}
I20260924 03:15:23.320196 17276 tablet_service.cc:1467] Processing CreateTablet for tablet b3baf9872a2c4845b2679b8a40644bcc (DEFAULT_TABLE table=test-table [id=7a9fef0149e1424fbb24876d5d33364b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:23.321727 17276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3baf9872a2c4845b2679b8a40644bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.321707 17415 tablet_service.cc:1467] Processing CreateTablet for tablet b3baf9872a2c4845b2679b8a40644bcc (DEFAULT_TABLE table=test-table [id=7a9fef0149e1424fbb24876d5d33364b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:23.323102 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3baf9872a2c4845b2679b8a40644bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.324856 17142 tablet_service.cc:1467] Processing CreateTablet for tablet b3baf9872a2c4845b2679b8a40644bcc (DEFAULT_TABLE table=test-table [id=7a9fef0149e1424fbb24876d5d33364b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:23.326128 17142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3baf9872a2c4845b2679b8a40644bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.337659 17504 tablet_bootstrap.cc:492] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Bootstrap starting.
I20260924 03:15:23.338451 17505 tablet_bootstrap.cc:492] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: Bootstrap starting.
I20260924 03:15:23.339830 17506 tablet_bootstrap.cc:492] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Bootstrap starting.
I20260924 03:15:23.341351 17504 tablet_bootstrap.cc:654] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:23.342499 17506 tablet_bootstrap.cc:654] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:23.342502 17504 log.cc:824] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:23.343652 17506 log.cc:824] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:23.344892 17505 tablet_bootstrap.cc:654] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:23.345538 17504 tablet_bootstrap.cc:492] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: No bootstrap required, opened a new log
I20260924 03:15:23.345984 17504 ts_tablet_manager.cc:1397] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 03:15:23.346406 17505 log.cc:824] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:23.346498 17506 tablet_bootstrap.cc:492] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: No bootstrap required, opened a new log
I20260924 03:15:23.346789 17506 ts_tablet_manager.cc:1397] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260924 03:15:23.353467 17506 raft_consensus.cc:348] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.353966 17506 raft_consensus.cc:374] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.354123 17506 raft_consensus.cc:729] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450df1629adc458ba9d3cd8f3f6008fb, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.354758 17506 consensus_queue.cc:260] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.355823 17505 tablet_bootstrap.cc:492] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: No bootstrap required, opened a new log
I20260924 03:15:23.356240 17505 ts_tablet_manager.cc:1397] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.009s
I20260924 03:15:23.358783 17485 heartbeater.cc:503] Master 127.14.227.190:44129 was elected leader, sending a full tablet report...
I20260924 03:15:23.359509 17506 ts_tablet_manager.cc:1428] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260924 03:15:23.359306 17504 raft_consensus.cc:348] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.359738 17504 raft_consensus.cc:374] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.359859 17504 raft_consensus.cc:729] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17cb4c30ad8455e89135d44a5ea6c0b, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.360462 17504 consensus_queue.cc:260] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.362030 17504 ts_tablet_manager.cc:1428] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260924 03:15:23.362258 17346 heartbeater.cc:503] Master 127.14.227.190:44129 was elected leader, sending a full tablet report...
I20260924 03:15:23.362437 17510 raft_consensus.cc:482] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:23.362663 17510 raft_consensus.cc:504] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.367054 17505 raft_consensus.cc:348] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.367498 17505 raft_consensus.cc:374] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.367642 17505 raft_consensus.cc:729] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e233f82279040c382b829f3ec3edc92, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.368454 17510 leader_election.cc:288] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953), 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783)
I20260924 03:15:23.368436 17505 consensus_queue.cc:260] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.371341 17505 ts_tablet_manager.cc:1428] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260924 03:15:23.372939 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.364708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45953 (local address 127.14.227.131:36765)
0924 03:15:23.365271 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.365291 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.365398 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.365977 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.365983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.366010 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.366342 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.366364 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.368111 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.368118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.370402 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.370419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.372208 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.372214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.372399 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.372431 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.372530 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.372657 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":29}
I20260924 03:15:23.373075 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.365289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:36765 (local address 127.14.227.130:45953)
0924 03:15:23.365471 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.365478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.365495 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:15:23.365553 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.365558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.365708 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.365800 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.366616 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.367916 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.370644 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.372042 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.372237 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.372327 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.372591 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.372750 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:15:23.374203 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3baf9872a2c4845b2679b8a40644bcc" candidate_uuid: "450df1629adc458ba9d3cd8f3f6008fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" is_pre_election: true
I20260924 03:15:23.374769 17296 raft_consensus.cc:2454] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450df1629adc458ba9d3cd8f3f6008fb in term 0.
I20260924 03:15:23.374889 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.366820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:33333 (local address 127.14.227.129:39783)
0924 03:15:23.366993 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.366998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.367169 (+ 171us) server_negotiation.cc:407] Connection header received
0924 03:15:23.367454 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.367461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.367572 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.367651 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.369268 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.370624 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.372800 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.374023 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.374201 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.374266 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.374547 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.374669 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:15:23.375116 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.364820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39783 (local address 127.14.227.131:33333)
0924 03:15:23.367057 (+ 2237us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.367079 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.367279 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.367860 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.367865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.367893 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.369019 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.369037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.370788 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.370796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.372643 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.372654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.374218 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.374224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.374394 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.374417 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.374517 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.374632 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2062,"mutex_wait_us":597,"thread_start_us":3349,"threads_started":1}
I20260924 03:15:23.375590 17369 leader_election.cc:302] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450df1629adc458ba9d3cd8f3f6008fb, b17cb4c30ad8455e89135d44a5ea6c0b; no voters:
I20260924 03:15:23.375982 17510 raft_consensus.cc:2781] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:23.376129 17510 raft_consensus.cc:482] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:23.376230 17510 raft_consensus.cc:3037] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.376066 17162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3baf9872a2c4845b2679b8a40644bcc" candidate_uuid: "450df1629adc458ba9d3cd8f3f6008fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e233f82279040c382b829f3ec3edc92" is_pre_election: true
I20260924 03:15:23.376569 17162 raft_consensus.cc:2454] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450df1629adc458ba9d3cd8f3f6008fb in term 0.
I20260924 03:15:23.379065 17510 raft_consensus.cc:504] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.379750 17510 leader_election.cc:288] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [CANDIDATE]: Term 1 election: Requested vote from peers b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953), 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783)
I20260924 03:15:23.380116 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3baf9872a2c4845b2679b8a40644bcc" candidate_uuid: "450df1629adc458ba9d3cd8f3f6008fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b"
I20260924 03:15:23.380224 17162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3baf9872a2c4845b2679b8a40644bcc" candidate_uuid: "450df1629adc458ba9d3cd8f3f6008fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e233f82279040c382b829f3ec3edc92"
I20260924 03:15:23.380321 17296 raft_consensus.cc:3037] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.380446 17162 raft_consensus.cc:3037] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.382714 17162 raft_consensus.cc:2454] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450df1629adc458ba9d3cd8f3f6008fb in term 1.
I20260924 03:15:23.383075 17296 raft_consensus.cc:2454] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450df1629adc458ba9d3cd8f3f6008fb in term 1.
I20260924 03:15:23.383101 17366 leader_election.cc:302] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450df1629adc458ba9d3cd8f3f6008fb, 6e233f82279040c382b829f3ec3edc92; no voters:
I20260924 03:15:23.383404 17510 raft_consensus.cc:2781] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:23.384045 17510 raft_consensus.cc:686] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 1 LEADER]: Becoming Leader. State: Replica: 450df1629adc458ba9d3cd8f3f6008fb, State: Running, Role: LEADER
I20260924 03:15:23.384635 17510 consensus_queue.cc:237] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.393055 17000 catalog_manager.cc:5495] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131). New cstate: current_term: 1 leader_uuid: "450df1629adc458ba9d3cd8f3f6008fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } health_report { overall_health: HEALTHY } } }
Sep 24 03:15:23 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219721, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:23.442365 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.428938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.14.227.190:44129)
0924 03:15:23.429120 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.429124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.429723 (+ 599us) server_negotiation.cc:407] Connection header received
0924 03:15:23.429904 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.429909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.430059 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.430159 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:23.431350 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.432163 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.433376 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.433801 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.433899 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.437262 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:23.437267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:23.437270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:23.437298 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:23.439814 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:23.440491 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:23.440495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:23.440497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:23.440610 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:23.441009 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:23.441012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:23.441014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:23.441275 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:23.441379 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.441829 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.442117 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":75}
I20260924 03:15:23.442387 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.428931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44129 (local address 127.0.0.1:45642)
0924 03:15:23.429609 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.429625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.429746 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.430212 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.430216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.430928 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:23.431194 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.431202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.432283 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.432286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.433248 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.433258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.434140 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.434146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.434259 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.434907 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:23.434925 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:23.437103 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:23.439940 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:23.439950 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:23.439956 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:23.440392 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:23.440753 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:23.440757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:23.440760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:23.440911 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:23.441399 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:23.441406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:23.441557 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.441724 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.441937 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":356,"threads_started":1}
I20260924 03:15:23.447654 17000 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:45642:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:15:23.448701 17000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 03:15:23.450263 17347 tablet.cc:2369] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:23.455994 17142 tablet_service.cc:1467] Processing CreateTablet for tablet 803d9041a1d04a638106a6b595b60175 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f85e67d8edc34c89bfb53e7efb176c11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:15:23.456156 17276 tablet_service.cc:1467] Processing CreateTablet for tablet 803d9041a1d04a638106a6b595b60175 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f85e67d8edc34c89bfb53e7efb176c11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:15:23.456854 17142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803d9041a1d04a638106a6b595b60175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.456899 17276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803d9041a1d04a638106a6b595b60175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.461540 17415 tablet_service.cc:1467] Processing CreateTablet for tablet 803d9041a1d04a638106a6b595b60175 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f85e67d8edc34c89bfb53e7efb176c11]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:15:23.462595 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803d9041a1d04a638106a6b595b60175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.462656 17504 tablet_bootstrap.cc:492] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: Bootstrap starting.
I20260924 03:15:23.463974 17504 tablet_bootstrap.cc:654] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:23.464712 17505 tablet_bootstrap.cc:492] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: Bootstrap starting.
I20260924 03:15:23.466348 17505 tablet_bootstrap.cc:654] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: Neither blocks nor log segments found. Creating new log.
W20260924 03:15:23.467787 17486 tablet.cc:2369] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:23.468681 17505 tablet_bootstrap.cc:492] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: No bootstrap required, opened a new log
I20260924 03:15:23.468813 17505 ts_tablet_manager.cc:1397] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 03:15:23.469555 17505 raft_consensus.cc:348] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.469784 17505 raft_consensus.cc:374] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.469874 17505 raft_consensus.cc:729] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e233f82279040c382b829f3ec3edc92, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.470254 17506 tablet_bootstrap.cc:492] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: Bootstrap starting.
I20260924 03:15:23.470116 17505 consensus_queue.cc:260] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.470733 17505 ts_tablet_manager.cc:1428] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:15:23.470996 17504 tablet_bootstrap.cc:492] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: No bootstrap required, opened a new log
I20260924 03:15:23.470839 17513 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.471143 17504 ts_tablet_manager.cc:1397] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 03:15:23.471192 17513 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 03:15:23.471508 17208 tablet.cc:2369] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:23.471909 17506 tablet_bootstrap.cc:654] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:23.471909 17504 raft_consensus.cc:348] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.472227 17504 raft_consensus.cc:374] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.472321 17504 raft_consensus.cc:729] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17cb4c30ad8455e89135d44a5ea6c0b, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.472559 17504 consensus_queue.cc:260] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.473049 17511 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.473343 17504 ts_tablet_manager.cc:1428] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:15:23.473469 17511 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:23.474821 17506 tablet_bootstrap.cc:492] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: No bootstrap required, opened a new log
I20260924 03:15:23.474973 17506 ts_tablet_manager.cc:1397] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260924 03:15:23.475749 17506 raft_consensus.cc:348] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.475958 17506 raft_consensus.cc:374] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.476050 17506 raft_consensus.cc:729] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450df1629adc458ba9d3cd8f3f6008fb, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.476282 17506 consensus_queue.cc:260] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.476667 17510 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.476850 17506 ts_tablet_manager.cc:1428] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 03:15:23.476956 17510 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:23.495193 17513 raft_consensus.cc:482] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:23.495487 17513 raft_consensus.cc:504] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.497330 17513 leader_election.cc:288] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953), 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433)
I20260924 03:15:23.504505 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.497770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45953 (local address 127.14.227.129:49487)
0924 03:15:23.498379 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.498395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.498560 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.499042 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.499045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.499073 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.499405 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.499415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.500686 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.500689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.502685 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.502699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.503941 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.503945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.504060 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.504082 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.504151 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.504234 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":26,"thread_start_us":327,"threads_started":1}
I20260924 03:15:23.504611 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.497936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:49487 (local address 127.14.227.130:45953)
0924 03:15:23.498194 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.498200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.498506 (+ 306us) server_negotiation.cc:407] Connection header received
0924 03:15:23.498787 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.498795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.498900 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.499034 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.499568 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.500578 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.502825 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.503751 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.503910 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.503976 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.504211 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.504336 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 03:15:23.505326 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.498752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:38083 (local address 127.14.227.131:41433)
0924 03:15:23.498930 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.498936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.499845 (+ 909us) server_negotiation.cc:407] Connection header received
0924 03:15:23.499950 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.499957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.500032 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.500122 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.500581 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.501286 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.503373 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.504491 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.504652 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.504732 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.505030 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.505157 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:15:23.505494 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.498563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:41433 (local address 127.14.227.129:38083)
0924 03:15:23.499744 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.499757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.499854 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.500137 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.500140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.500157 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.500432 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.500439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.501424 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.501429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.503240 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.503250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.504704 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.504712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.504858 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.504889 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.504989 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.505138 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"mutex_wait_us":20,"thread_start_us":853,"threads_started":1}
I20260924 03:15:23.505544 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803d9041a1d04a638106a6b595b60175" candidate_uuid: "6e233f82279040c382b829f3ec3edc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" is_pre_election: true
I20260924 03:15:23.505846 17296 raft_consensus.cc:2454] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e233f82279040c382b829f3ec3edc92 in term 0.
I20260924 03:15:23.506392 17090 leader_election.cc:302] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e233f82279040c382b829f3ec3edc92, b17cb4c30ad8455e89135d44a5ea6c0b; no voters:
I20260924 03:15:23.506798 17513 raft_consensus.cc:2781] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:23.506915 17513 raft_consensus.cc:482] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:23.507019 17513 raft_consensus.cc:3037] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.506892 17439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803d9041a1d04a638106a6b595b60175" candidate_uuid: "6e233f82279040c382b829f3ec3edc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450df1629adc458ba9d3cd8f3f6008fb" is_pre_election: true
I20260924 03:15:23.507337 17439 raft_consensus.cc:2454] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e233f82279040c382b829f3ec3edc92 in term 0.
I20260924 03:15:23.509339 17513 raft_consensus.cc:504] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.509912 17513 leader_election.cc:288] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [CANDIDATE]: Term 1 election: Requested vote from peers b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953), 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433)
I20260924 03:15:23.510339 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803d9041a1d04a638106a6b595b60175" candidate_uuid: "6e233f82279040c382b829f3ec3edc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b"
I20260924 03:15:23.510492 17439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803d9041a1d04a638106a6b595b60175" candidate_uuid: "6e233f82279040c382b829f3ec3edc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450df1629adc458ba9d3cd8f3f6008fb"
I20260924 03:15:23.510560 17296 raft_consensus.cc:3037] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.510672 17439 raft_consensus.cc:3037] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.513548 17296 raft_consensus.cc:2454] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e233f82279040c382b829f3ec3edc92 in term 1.
I20260924 03:15:23.513550 17439 raft_consensus.cc:2454] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e233f82279040c382b829f3ec3edc92 in term 1.
I20260924 03:15:23.513885 17090 leader_election.cc:302] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e233f82279040c382b829f3ec3edc92, b17cb4c30ad8455e89135d44a5ea6c0b; no voters:
I20260924 03:15:23.514195 17513 raft_consensus.cc:2781] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:23.514581 17513 raft_consensus.cc:686] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [term 1 LEADER]: Becoming Leader. State: Replica: 6e233f82279040c382b829f3ec3edc92, State: Running, Role: LEADER
I20260924 03:15:23.515054 17513 consensus_queue.cc:237] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.517777 17527 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: TxnStatusTablet state changed. Reason: New leader 6e233f82279040c382b829f3ec3edc92. Latest consensus state: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.518180 17527 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:15:23.518887 17529 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:15:23.519150 17529 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:15:23.519731 17000 catalog_manager.cc:5495] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e233f82279040c382b829f3ec3edc92 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } health_report { overall_health: UNKNOWN } } }
I20260924 03:15:23.551111 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.544496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.14.227.129:39783)
0924 03:15:23.544667 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.544674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.544704 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:15:23.544980 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.544984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.545067 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.545187 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:23.545613 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.546445 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.548368 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.549038 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.549180 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.549729 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.550422 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.550583 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.550704 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.550782 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:15:23.551149 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.544340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39783 (local address 127.0.0.1:56406)
0924 03:15:23.544594 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.544614 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.544747 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.545248 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.545253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.545266 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:23.545459 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.545470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.546566 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.546571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.548212 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.548226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.549210 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.549216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.549469 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.549541 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.550549 (+ 1008us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.550581 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.550674 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.550825 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 03:15:23.560225 17439 raft_consensus.cc:1261] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Refusing update from remote peer 6e233f82279040c382b829f3ec3edc92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:23.560312 17296 raft_consensus.cc:1261] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Refusing update from remote peer 6e233f82279040c382b829f3ec3edc92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:23.561317 17513 consensus_queue.cc:1035] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:23.562002 17527 consensus_queue.cc:1035] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:23.576572 17510 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: TxnStatusTablet state changed. Reason: New leader 6e233f82279040c382b829f3ec3edc92. Latest consensus state: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.576900 17510 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:23.579741 17532 mvcc.cc:205] Tried to move back new op lower bound from 7332739987692457984 to 7332739987521355776. Current Snapshot: MvccSnapshot[applied={T|T < 7332739987692457984}]
I20260924 03:15:23.580615 17527 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.580852 17527 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:15:23.581172 17533 mvcc.cc:205] Tried to move back new op lower bound from 7332739987692457984 to 7332739987521355776. Current Snapshot: MvccSnapshot[applied={T|T < 7332739987692457984}]
I20260924 03:15:23.589028 17510 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.589335 17510 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P 450df1629adc458ba9d3cd8f3f6008fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:23.594937 17511 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: TxnStatusTablet state changed. Reason: New leader 6e233f82279040c382b829f3ec3edc92. Latest consensus state: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.595263 17511 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:23.596412 17513 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.596725 17513 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P 6e233f82279040c382b829f3ec3edc92: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:15:23.610411 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.599615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.14.227.131:41433)
0924 03:15:23.599823 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.599832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.600080 (+ 248us) server_negotiation.cc:407] Connection header received
0924 03:15:23.600352 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.600360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.600501 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.600681 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:23.601393 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.602924 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.605653 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.606570 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.606719 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.607488 (+ 769us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.608354 (+ 866us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.608516 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.609472 (+ 956us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.609595 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:15:23.612984 17511 tablet_replica.cc:439] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } } }
I20260924 03:15:23.613257 17511 tablet_replica.cc:442] T 803d9041a1d04a638106a6b595b60175 P b17cb4c30ad8455e89135d44a5ea6c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:15:23.627166 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.621479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:50685 (local address 127.14.227.129:39783)
0924 03:15:23.621667 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.621684 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.621774 (+ 90us) server_negotiation.cc:407] Connection header received
0924 03:15:23.621924 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.621930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.622055 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.622174 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:23.622604 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.623781 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.625027 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.625753 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.625874 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.626363 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.626493 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.626579 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.626772 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.626850 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:15:23.627167 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.621471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39783 (local address 127.14.227.131:50685)
0924 03:15:23.621693 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.621708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.621800 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.622209 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.622217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.622230 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:23.622421 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.622433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.623903 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.623907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.624896 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.624907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.625885 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.625889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.626105 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.626180 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.626623 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.626638 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.626732 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.626888 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:15:23.636396 17296 raft_consensus.cc:1261] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Refusing update from remote peer 450df1629adc458ba9d3cd8f3f6008fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:23.636642 17162 raft_consensus.cc:1261] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Refusing update from remote peer 450df1629adc458ba9d3cd8f3f6008fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:23.637045 17514 consensus_queue.cc:1035] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:23.637434 17510 consensus_queue.cc:1035] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:23.675112 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.668959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:36113 (local address 127.14.227.131:41433)
0924 03:15:23.669148 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.669154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.669425 (+ 271us) server_negotiation.cc:407] Connection header received
0924 03:15:23.669592 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.669597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.669668 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.669761 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:23.670312 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.671413 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.672895 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.673540 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.673662 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.674353 (+ 691us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.674488 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.674551 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.674781 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.674860 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:15:23.675443 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.668687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:41433 (local address 127.14.227.129:36113)
0924 03:15:23.669307 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.669325 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.669466 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.669787 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.669792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.669804 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:23.670158 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.670168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.671549 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.671554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.672753 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.672765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.673848 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.673854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.674065 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.674140 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.674636 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.674647 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.674740 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.674891 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":469,"threads_started":1}
I20260924 03:15:23.732138 16999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:23.732501 16999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:23.736651 16999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: Sending DeleteTablet for 3 replicas of tablet b3baf9872a2c4845b2679b8a40644bcc
I20260924 03:15:23.737787 17276 tablet_service.cc:1514] Processing DeleteTablet for tablet b3baf9872a2c4845b2679b8a40644bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:23 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:57734
I20260924 03:15:23.738394 17142 tablet_service.cc:1514] Processing DeleteTablet for tablet b3baf9872a2c4845b2679b8a40644bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:23 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56398
I20260924 03:15:23.739301 17415 tablet_service.cc:1514] Processing DeleteTablet for tablet b3baf9872a2c4845b2679b8a40644bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:23 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38604
I20260924 03:15:23.742202 16999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:23.742870 16999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:23.751824 17550 tablet_replica.cc:330] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: stopping tablet replica
I20260924 03:15:23.752437 17550 raft_consensus.cc:2229] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:23.752908 17550 raft_consensus.cc:2258] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:23.754491 17276 tablet_service.cc:1467] Processing CreateTablet for tablet f7c943fe46c146478f2345bb00673e60 (DEFAULT_TABLE table=test-table [id=46f187b8e42e493facc5def0c5b0f2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:23.754695 17141 tablet_service.cc:1467] Processing CreateTablet for tablet f7c943fe46c146478f2345bb00673e60 (DEFAULT_TABLE table=test-table [id=46f187b8e42e493facc5def0c5b0f2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:23.754734 17414 tablet_service.cc:1467] Processing CreateTablet for tablet f7c943fe46c146478f2345bb00673e60 (DEFAULT_TABLE table=test-table [id=46f187b8e42e493facc5def0c5b0f2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:23.755213 17276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c943fe46c146478f2345bb00673e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.755272 17141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c943fe46c146478f2345bb00673e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.755317 17414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c943fe46c146478f2345bb00673e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:23.761991 17506 tablet_bootstrap.cc:492] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: Bootstrap starting.
I20260924 03:15:23.763563 17550 ts_tablet_manager.cc:1905] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:23.763703 17506 tablet_bootstrap.cc:654] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:23.768072 17505 tablet_bootstrap.cc:492] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: Bootstrap starting.
I20260924 03:15:23.768442 17504 tablet_bootstrap.cc:492] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: Bootstrap starting.
I20260924 03:15:23.769755 17505 tablet_bootstrap.cc:654] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:23.770315 17504 tablet_bootstrap.cc:654] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:23.779953 17505 tablet_bootstrap.cc:492] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: No bootstrap required, opened a new log
I20260924 03:15:23.780169 17505 ts_tablet_manager.cc:1397] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 03:15:23.781139 17505 raft_consensus.cc:348] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.781451 17505 raft_consensus.cc:374] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.781534 17505 raft_consensus.cc:729] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e233f82279040c382b829f3ec3edc92, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.782125 17505 consensus_queue.cc:260] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.783247 17505 ts_tablet_manager.cc:1428] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:23.785063 17504 tablet_bootstrap.cc:492] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: No bootstrap required, opened a new log
I20260924 03:15:23.785393 17504 ts_tablet_manager.cc:1397] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20260924 03:15:23.786499 17504 raft_consensus.cc:348] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.786841 17504 raft_consensus.cc:374] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.786955 17504 raft_consensus.cc:729] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17cb4c30ad8455e89135d44a5ea6c0b, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.787285 17504 consensus_queue.cc:260] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.787925 17504 ts_tablet_manager.cc:1428] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 03:15:23.794301 17552 tablet_replica.cc:330] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: stopping tablet replica
I20260924 03:15:23.794818 17552 raft_consensus.cc:2229] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:15:23.795599 17552 raft_consensus.cc:2258] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:23.796022 17506 tablet_bootstrap.cc:492] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: No bootstrap required, opened a new log
I20260924 03:15:23.796203 17506 ts_tablet_manager.cc:1397] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.010s
I20260924 03:15:23.796319 17551 tablet_replica.cc:330] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: stopping tablet replica
I20260924 03:15:23.796743 17551 raft_consensus.cc:2229] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:23.796931 17550 ts_tablet_manager.cc:1918] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:23.797049 17551 raft_consensus.cc:2258] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:23.797219 17550 log.cc:1196] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/b3baf9872a2c4845b2679b8a40644bcc
I20260924 03:15:23.797904 17550 ts_tablet_manager.cc:1939] T b3baf9872a2c4845b2679b8a40644bcc P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting consensus metadata
I20260924 03:15:23.798218 17506 raft_consensus.cc:348] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.798466 17506 raft_consensus.cc:374] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:23.798552 17506 raft_consensus.cc:729] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450df1629adc458ba9d3cd8f3f6008fb, State: Initialized, Role: FOLLOWER
I20260924 03:15:23.799149 17506 consensus_queue.cc:260] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.799315 17551 ts_tablet_manager.cc:1905] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:23.799706 17506 ts_tablet_manager.cc:1428] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:23.800010 17552 ts_tablet_manager.cc:1905] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:23.801363 16987 catalog_manager.cc:4841] TS b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953): tablet b3baf9872a2c4845b2679b8a40644bcc (table test-table [id=7a9fef0149e1424fbb24876d5d33364b]) successfully deleted
I20260924 03:15:23.806921 17551 ts_tablet_manager.cc:1918] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:23.807116 17551 log.cc:1196] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/b3baf9872a2c4845b2679b8a40644bcc
I20260924 03:15:23.807189 17552 ts_tablet_manager.cc:1918] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:23.807345 17552 log.cc:1196] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/b3baf9872a2c4845b2679b8a40644bcc
I20260924 03:15:23.807737 17551 ts_tablet_manager.cc:1939] T b3baf9872a2c4845b2679b8a40644bcc P 6e233f82279040c382b829f3ec3edc92: Deleting consensus metadata
I20260924 03:15:23.807835 17552 ts_tablet_manager.cc:1939] T b3baf9872a2c4845b2679b8a40644bcc P 450df1629adc458ba9d3cd8f3f6008fb: Deleting consensus metadata
I20260924 03:15:23.809399 16987 catalog_manager.cc:4841] TS 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433): tablet b3baf9872a2c4845b2679b8a40644bcc (table test-table [id=7a9fef0149e1424fbb24876d5d33364b]) successfully deleted
I20260924 03:15:23.809407 16984 catalog_manager.cc:4841] TS 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783): tablet b3baf9872a2c4845b2679b8a40644bcc (table test-table [id=7a9fef0149e1424fbb24876d5d33364b]) successfully deleted
I20260924 03:15:23.855937 17511 raft_consensus.cc:482] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:23.856242 17511 raft_consensus.cc:504] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.858155 17511 leader_election.cc:288] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783), 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433)
I20260924 03:15:23.878687 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.858986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:44601 (local address 127.14.227.129:39783)
0924 03:15:23.859200 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.859206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.869641 (+ 10435us) server_negotiation.cc:407] Connection header received
0924 03:15:23.870217 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.870224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.870301 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.870369 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.871189 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.872429 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.875841 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.877351 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.877521 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.877605 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.878200 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.878351 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:15:23.879676 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.858968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:46413 (local address 127.14.227.131:41433)
0924 03:15:23.859190 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.859197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.871965 (+ 12768us) server_negotiation.cc:407] Connection header received
0924 03:15:23.872383 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.872390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.872467 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.872529 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:23.873407 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.874581 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.877109 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.878498 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.878619 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.878684 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.879223 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.879366 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:15:23.880136 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.858478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39783 (local address 127.14.227.130:44601)
0924 03:15:23.869522 (+ 11044us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.869546 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.870023 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.870575 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.870581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.870611 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.870979 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.870991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.872620 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.872626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.875660 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.875680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.877789 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.877795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.878004 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.878038 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.879396 (+ 1358us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.879540 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10877,"mutex_wait_us":28,"thread_start_us":11185,"threads_started":1}
I20260924 03:15:23.880517 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.858602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:41433 (local address 127.14.227.130:46413)
0924 03:15:23.871858 (+ 13256us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.871877 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.872211 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.872725 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.872730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.872757 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:23.873212 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.873224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.874809 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.874814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.876902 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.876918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.878868 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.878874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.879035 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.879067 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.879914 (+ 847us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.880095 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13128,"mutex_wait_us":41,"thread_start_us":13383,"threads_started":1}
I20260924 03:15:23.881085 17162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c943fe46c146478f2345bb00673e60" candidate_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e233f82279040c382b829f3ec3edc92" is_pre_election: true
I20260924 03:15:23.881150 17438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c943fe46c146478f2345bb00673e60" candidate_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450df1629adc458ba9d3cd8f3f6008fb" is_pre_election: true
I20260924 03:15:23.881363 17162 raft_consensus.cc:2454] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b17cb4c30ad8455e89135d44a5ea6c0b in term 0.
I20260924 03:15:23.881395 17438 raft_consensus.cc:2454] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b17cb4c30ad8455e89135d44a5ea6c0b in term 0.
I20260924 03:15:23.881911 17230 leader_election.cc:302] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450df1629adc458ba9d3cd8f3f6008fb, b17cb4c30ad8455e89135d44a5ea6c0b; no voters:
I20260924 03:15:23.882426 17511 raft_consensus.cc:2781] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:23.882552 17511 raft_consensus.cc:482] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:23.882649 17511 raft_consensus.cc:3037] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.885696 17511 raft_consensus.cc:504] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.886494 17511 leader_election.cc:288] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [CANDIDATE]: Term 1 election: Requested vote from peers 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783), 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433)
I20260924 03:15:23.887178 17162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c943fe46c146478f2345bb00673e60" candidate_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e233f82279040c382b829f3ec3edc92"
I20260924 03:15:23.887182 17438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c943fe46c146478f2345bb00673e60" candidate_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450df1629adc458ba9d3cd8f3f6008fb"
I20260924 03:15:23.887385 17438 raft_consensus.cc:3037] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.887387 17162 raft_consensus.cc:3037] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:23.890565 17162 raft_consensus.cc:2454] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b17cb4c30ad8455e89135d44a5ea6c0b in term 1.
I20260924 03:15:23.891001 17227 leader_election.cc:302] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e233f82279040c382b829f3ec3edc92, b17cb4c30ad8455e89135d44a5ea6c0b; no voters:
I20260924 03:15:23.891299 17511 raft_consensus.cc:2781] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:23.892387 17438 raft_consensus.cc:2454] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b17cb4c30ad8455e89135d44a5ea6c0b in term 1.
I20260924 03:15:23.893216 17511 raft_consensus.cc:686] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 LEADER]: Becoming Leader. State: Replica: b17cb4c30ad8455e89135d44a5ea6c0b, State: Running, Role: LEADER
I20260924 03:15:23.893635 17511 consensus_queue.cc:237] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:23.897447 16998 catalog_manager.cc:5495] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b reported cstate change: term changed from 0 to 1, leader changed from <none> to b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130). New cstate: current_term: 1 leader_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } health_report { overall_health: UNKNOWN } } }
I20260924 03:15:23.945128 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.938329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.14.227.130:45953)
0924 03:15:23.938493 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.938499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.938518 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:15:23.938615 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.938621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.938728 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.938889 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:23.939311 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.940473 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.942426 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.943109 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.943230 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.943737 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.944454 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.944607 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.944726 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.944814 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:15:23.964003 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.956900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:40487 (local address 127.14.227.129:39783)
0924 03:15:23.957083 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:23.957089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:23.957708 (+ 619us) server_negotiation.cc:407] Connection header received
0924 03:15:23.957891 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:23.957897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:23.958033 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:23.958130 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:23.958705 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.959853 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.961888 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.962563 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.962701 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.963209 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.963365 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.963461 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:23.963609 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:15:23.963700 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:15:23.964097 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:23.956926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39783 (local address 127.14.227.130:40487)
0924 03:15:23.957585 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:23.957604 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:23.957746 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:23.958175 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:23.958183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:23.958196 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:23.958528 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.958539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.959994 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.959999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.961733 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:23.961748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:23.962754 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:23.962760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:23.962945 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:23.963012 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:23.963474 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:23.963485 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:15:23.963570 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:15:23.963720 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":329,"threads_started":1}
I20260924 03:15:23.974897 17162 raft_consensus.cc:1261] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Refusing update from remote peer b17cb4c30ad8455e89135d44a5ea6c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:23.975626 17558 consensus_queue.cc:1035] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:23.977596 17438 raft_consensus.cc:1261] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Refusing update from remote peer b17cb4c30ad8455e89135d44a5ea6c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:23.978142 17558 consensus_queue.cc:1035] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:23.992028 17534 mvcc.cc:205] Tried to move back new op lower bound from 7332739989394030592 to 7332739989072752640. Current Snapshot: MvccSnapshot[applied={T|T < 7332739989394030592}]
I20260924 03:15:24.023649 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:24.016336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45953 (local address 127.14.227.129:59675)
0924 03:15:24.016528 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:24.016561 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:24.016674 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:24.017522 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:24.017527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:24.017539 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:24.017753 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:24.017766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:24.019159 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:24.019164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:24.020806 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:24.020821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:24.022401 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:24.022407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:24.022612 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:24.022662 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:24.023105 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:24.023116 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:15:24.023187 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:15:24.023326 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:15:24.024345 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:24.016709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:59675 (local address 127.14.227.130:45953)
0924 03:15:24.016892 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:24.016898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:24.016915 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:15:24.016987 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:24.016992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:24.017077 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:24.017168 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:24.017918 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:24.019004 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:24.021486 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:24.022166 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:24.022281 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:24.022835 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:24.022978 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:24.023058 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:24.024006 (+ 948us) server_negotiation.cc:299] Negotiation successful
0924 03:15:24.024085 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 03:15:24.026340 17534 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:15:25.641875 16998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:25.642172 16998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:25.646078 16998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: Sending DeleteTablet for 3 replicas of tablet f7c943fe46c146478f2345bb00673e60
I20260924 03:15:25.646890 17142 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c943fe46c146478f2345bb00673e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:25 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56398
I20260924 03:15:25.646981 17276 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c943fe46c146478f2345bb00673e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:25 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:57734
I20260924 03:15:25.647075 17415 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c943fe46c146478f2345bb00673e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:25 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38604
I20260924 03:15:25.659415 17589 tablet_replica.cc:330] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: stopping tablet replica
I20260924 03:15:25.659739 17589 raft_consensus.cc:2229] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:15:25.660148 17588 tablet_replica.cc:330] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: stopping tablet replica
I20260924 03:15:25.660307 17590 tablet_replica.cc:330] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: stopping tablet replica
I20260924 03:15:25.660522 17590 raft_consensus.cc:2229] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:25.660550 17588 raft_consensus.cc:2229] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:25.660792 17590 raft_consensus.cc:2258] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:25.660796 17589 raft_consensus.cc:2258] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:25.660833 17588 raft_consensus.cc:2258] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:25.661868 17589 ts_tablet_manager.cc:1905] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:25.661931 17588 ts_tablet_manager.cc:1905] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:25.662104 17590 ts_tablet_manager.cc:1905] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:25.668659 17590 ts_tablet_manager.cc:1918] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:15:25.668886 17590 log.cc:1196] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/f7c943fe46c146478f2345bb00673e60
I20260924 03:15:25.669044 17588 ts_tablet_manager.cc:1918] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:15:25.669256 17588 log.cc:1196] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/f7c943fe46c146478f2345bb00673e60
I20260924 03:15:25.669338 17589 ts_tablet_manager.cc:1918] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:15:25.669466 17590 ts_tablet_manager.cc:1939] T f7c943fe46c146478f2345bb00673e60 P 450df1629adc458ba9d3cd8f3f6008fb: Deleting consensus metadata
I20260924 03:15:25.669526 17589 log.cc:1196] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/f7c943fe46c146478f2345bb00673e60
I20260924 03:15:25.669836 17588 ts_tablet_manager.cc:1939] T f7c943fe46c146478f2345bb00673e60 P 6e233f82279040c382b829f3ec3edc92: Deleting consensus metadata
I20260924 03:15:25.669975 17589 ts_tablet_manager.cc:1939] T f7c943fe46c146478f2345bb00673e60 P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting consensus metadata
I20260924 03:15:25.670867 16987 catalog_manager.cc:4841] TS 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433): tablet f7c943fe46c146478f2345bb00673e60 (table test-table [id=46f187b8e42e493facc5def0c5b0f2ed]) successfully deleted
I20260924 03:15:25.671173 16987 catalog_manager.cc:4841] TS b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953): tablet f7c943fe46c146478f2345bb00673e60 (table test-table [id=46f187b8e42e493facc5def0c5b0f2ed]) successfully deleted
I20260924 03:15:25.671299 16984 catalog_manager.cc:4841] TS 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783): tablet f7c943fe46c146478f2345bb00673e60 (table test-table [id=46f187b8e42e493facc5def0c5b0f2ed]) successfully deleted
Sep 24 03:15:25 dist-test-slave-vd0p krb5kdc[16955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:15:25 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:25.705075 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:25.689510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.14.227.190:44129)
0924 03:15:25.690131 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:25.690139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:25.690166 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:15:25.690409 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:25.690413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:25.690473 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:25.690584 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:25.691981 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:25.692712 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:25.693923 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:25.694393 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:25.694525 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:25.698782 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:25.698789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:25.698794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:25.698836 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:25.702611 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:25.703203 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:25.703209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:25.703212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:25.703350 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:25.703664 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:25.703668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:25.703672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:25.703997 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:25.704227 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:25.704477 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 03:15:25.704718 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":303,"threads_started":1}
I20260924 03:15:25.708559 16998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:25.709203 16998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:25.715427 17142 tablet_service.cc:1467] Processing CreateTablet for tablet a3c57019d2fe44678774329c48ee7d22 (DEFAULT_TABLE table=test-table [id=baf466dd62ff40d58558fc5ca76ccaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:25.715618 17276 tablet_service.cc:1467] Processing CreateTablet for tablet a3c57019d2fe44678774329c48ee7d22 (DEFAULT_TABLE table=test-table [id=baf466dd62ff40d58558fc5ca76ccaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:25.715939 17142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c57019d2fe44678774329c48ee7d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:25.716027 17276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c57019d2fe44678774329c48ee7d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:25.716008 17415 tablet_service.cc:1467] Processing CreateTablet for tablet a3c57019d2fe44678774329c48ee7d22 (DEFAULT_TABLE table=test-table [id=baf466dd62ff40d58558fc5ca76ccaae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:25.716374 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c57019d2fe44678774329c48ee7d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:25.727927 17599 tablet_bootstrap.cc:492] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: Bootstrap starting.
I20260924 03:15:25.728528 17600 tablet_bootstrap.cc:492] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: Bootstrap starting.
I20260924 03:15:25.729020 17601 tablet_bootstrap.cc:492] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: Bootstrap starting.
I20260924 03:15:25.729885 17599 tablet_bootstrap.cc:654] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:25.730590 17600 tablet_bootstrap.cc:654] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:25.731492 17601 tablet_bootstrap.cc:654] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:25.732616 17599 tablet_bootstrap.cc:492] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: No bootstrap required, opened a new log
I20260924 03:15:25.732803 17599 ts_tablet_manager.cc:1397] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 03:15:25.733705 17600 tablet_bootstrap.cc:492] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: No bootstrap required, opened a new log
I20260924 03:15:25.733708 17601 tablet_bootstrap.cc:492] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: No bootstrap required, opened a new log
I20260924 03:15:25.733639 17599 raft_consensus.cc:348] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.733856 17601 ts_tablet_manager.cc:1397] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 03:15:25.733860 17600 ts_tablet_manager.cc:1397] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 03:15:25.733873 17599 raft_consensus.cc:374] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:25.733995 17599 raft_consensus.cc:729] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17cb4c30ad8455e89135d44a5ea6c0b, State: Initialized, Role: FOLLOWER
I20260924 03:15:25.734249 17599 consensus_queue.cc:260] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.734582 17600 raft_consensus.cc:348] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.734612 17601 raft_consensus.cc:348] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.734807 17599 ts_tablet_manager.cc:1428] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 03:15:25.734813 17600 raft_consensus.cc:374] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:25.734853 17601 raft_consensus.cc:374] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:25.734910 17600 raft_consensus.cc:729] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e233f82279040c382b829f3ec3edc92, State: Initialized, Role: FOLLOWER
I20260924 03:15:25.734936 17601 raft_consensus.cc:729] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450df1629adc458ba9d3cd8f3f6008fb, State: Initialized, Role: FOLLOWER
I20260924 03:15:25.735121 17600 consensus_queue.cc:260] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.735190 17601 consensus_queue.cc:260] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.735656 17600 ts_tablet_manager.cc:1428] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:15:25.736925 17601 ts_tablet_manager.cc:1428] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:25.817204 17540 raft_consensus.cc:482] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:25.817448 17540 raft_consensus.cc:504] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.818152 17540 leader_election.cc:288] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953), 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433)
I20260924 03:15:25.818755 17439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c57019d2fe44678774329c48ee7d22" candidate_uuid: "6e233f82279040c382b829f3ec3edc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450df1629adc458ba9d3cd8f3f6008fb" is_pre_election: true
I20260924 03:15:25.818733 17295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c57019d2fe44678774329c48ee7d22" candidate_uuid: "6e233f82279040c382b829f3ec3edc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" is_pre_election: true
I20260924 03:15:25.819043 17439 raft_consensus.cc:2454] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e233f82279040c382b829f3ec3edc92 in term 0.
I20260924 03:15:25.819044 17295 raft_consensus.cc:2454] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e233f82279040c382b829f3ec3edc92 in term 0.
I20260924 03:15:25.819520 17090 leader_election.cc:302] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450df1629adc458ba9d3cd8f3f6008fb, 6e233f82279040c382b829f3ec3edc92; no voters:
I20260924 03:15:25.819772 17540 raft_consensus.cc:2781] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:25.819896 17540 raft_consensus.cc:482] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:25.819998 17540 raft_consensus.cc:3037] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:25.822387 17540 raft_consensus.cc:504] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.822901 17540 leader_election.cc:288] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [CANDIDATE]: Term 1 election: Requested vote from peers b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953), 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433)
I20260924 03:15:25.823383 17295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c57019d2fe44678774329c48ee7d22" candidate_uuid: "6e233f82279040c382b829f3ec3edc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b"
I20260924 03:15:25.823432 17439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c57019d2fe44678774329c48ee7d22" candidate_uuid: "6e233f82279040c382b829f3ec3edc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "450df1629adc458ba9d3cd8f3f6008fb"
I20260924 03:15:25.823591 17295 raft_consensus.cc:3037] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:25.823624 17439 raft_consensus.cc:3037] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:25.826169 17439 raft_consensus.cc:2454] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e233f82279040c382b829f3ec3edc92 in term 1.
I20260924 03:15:25.826193 17295 raft_consensus.cc:2454] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e233f82279040c382b829f3ec3edc92 in term 1.
I20260924 03:15:25.826647 17090 leader_election.cc:302] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e233f82279040c382b829f3ec3edc92, b17cb4c30ad8455e89135d44a5ea6c0b; no voters:
I20260924 03:15:25.826944 17540 raft_consensus.cc:2781] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:25.827127 17540 raft_consensus.cc:686] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 1 LEADER]: Becoming Leader. State: Replica: 6e233f82279040c382b829f3ec3edc92, State: Running, Role: LEADER
I20260924 03:15:25.827436 17540 consensus_queue.cc:237] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.830335 17000 catalog_manager.cc:5495] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e233f82279040c382b829f3ec3edc92 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "6e233f82279040c382b829f3ec3edc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } health_report { overall_health: UNKNOWN } } }
I20260924 03:15:25.867520 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:25.862156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.14.227.129:39783)
0924 03:15:25.862735 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:25.862740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:25.862759 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:15:25.862812 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:25.862815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:25.862874 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:25.862958 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:25.863855 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:25.864596 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:25.865831 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:25.866254 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:25.866349 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:25.866759 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:25.866864 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:25.866930 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:25.867094 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 03:15:25.867196 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":350,"threads_started":1}
I20260924 03:15:25.880224 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:25.875786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:48029 (local address 127.14.227.129:39783)
0924 03:15:25.876027 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:25.876036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:25.876284 (+ 248us) server_negotiation.cc:407] Connection header received
0924 03:15:25.876408 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:25.876411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:25.876468 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:25.876538 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:25.876902 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:25.877581 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:25.878893 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:25.879276 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:25.879374 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:25.879707 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:25.879791 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:25.879854 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:25.879971 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 03:15:25.880024 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:15:25.880354 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:25.875655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39783 (local address 127.14.227.129:48029)
0924 03:15:25.876196 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:25.876213 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:25.876315 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:25.876562 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:25.876565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:25.876573 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:15:25.876779 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:25.876786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:25.877694 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:25.877697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:25.878772 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:25.878784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:25.879433 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:25.879437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:25.879554 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:25.879587 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:15:25.879869 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:15:25.879876 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:15:25.879939 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 03:15:25.880035 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":1,"thread_start_us":312,"threads_started":1}
I20260924 03:15:25.887984 17439 raft_consensus.cc:1261] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Refusing update from remote peer 6e233f82279040c382b829f3ec3edc92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:25.888043 17295 raft_consensus.cc:1261] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Refusing update from remote peer 6e233f82279040c382b829f3ec3edc92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:25.888557 17607 consensus_queue.cc:1035] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:25.888921 17540 consensus_queue.cc:1035] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:25.932294 17000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:25.932544 17000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:25.935384 17000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: Sending DeleteTablet for 3 replicas of tablet a3c57019d2fe44678774329c48ee7d22
I20260924 03:15:25.936112 17142 tablet_service.cc:1514] Processing DeleteTablet for tablet a3c57019d2fe44678774329c48ee7d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:25 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56398
I20260924 03:15:25.936280 17415 tablet_service.cc:1514] Processing DeleteTablet for tablet a3c57019d2fe44678774329c48ee7d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:25 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38604
I20260924 03:15:25.936432 17588 tablet_replica.cc:330] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: stopping tablet replica
I20260924 03:15:25.936558 17590 tablet_replica.cc:330] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: stopping tablet replica
I20260924 03:15:25.936640 17588 raft_consensus.cc:2229] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:15:25.936762 17590 raft_consensus.cc:2229] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:25.936957 17590 raft_consensus.cc:2258] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:25.937127 17588 raft_consensus.cc:2258] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:25.937435 17276 tablet_service.cc:1514] Processing DeleteTablet for tablet a3c57019d2fe44678774329c48ee7d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:25 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:57734
I20260924 03:15:25.937690 17589 tablet_replica.cc:330] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: stopping tablet replica
I20260924 03:15:25.937886 17589 raft_consensus.cc:2229] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:25.938015 17590 ts_tablet_manager.cc:1905] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:25.938099 17589 raft_consensus.cc:2258] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:25.938645 17588 ts_tablet_manager.cc:1905] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:25.943490 17000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:25.944283 17000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:25.944409 17588 ts_tablet_manager.cc:1918] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:25.944597 17588 log.cc:1196] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/a3c57019d2fe44678774329c48ee7d22
I20260924 03:15:25.944792 17590 ts_tablet_manager.cc:1918] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:25.945000 17590 log.cc:1196] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/a3c57019d2fe44678774329c48ee7d22
I20260924 03:15:25.945264 17588 ts_tablet_manager.cc:1939] T a3c57019d2fe44678774329c48ee7d22 P 6e233f82279040c382b829f3ec3edc92: Deleting consensus metadata
I20260924 03:15:25.945437 17590 ts_tablet_manager.cc:1939] T a3c57019d2fe44678774329c48ee7d22 P 450df1629adc458ba9d3cd8f3f6008fb: Deleting consensus metadata
I20260924 03:15:25.946291 17589 ts_tablet_manager.cc:1905] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:25.946431 16984 catalog_manager.cc:4841] TS 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783): tablet a3c57019d2fe44678774329c48ee7d22 (table test-table [id=baf466dd62ff40d58558fc5ca76ccaae]) successfully deleted
I20260924 03:15:25.947294 16987 catalog_manager.cc:4841] TS 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433): tablet a3c57019d2fe44678774329c48ee7d22 (table test-table [id=baf466dd62ff40d58558fc5ca76ccaae]) successfully deleted
I20260924 03:15:25.951961 17142 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3be79990e8425591742312f888e83a (DEFAULT_TABLE table=test-table [id=2daf91aa6b6348bf9e604dfa3886ab12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:25.952105 17276 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3be79990e8425591742312f888e83a (DEFAULT_TABLE table=test-table [id=2daf91aa6b6348bf9e604dfa3886ab12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:25.952287 17415 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3be79990e8425591742312f888e83a (DEFAULT_TABLE table=test-table [id=2daf91aa6b6348bf9e604dfa3886ab12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:25.952644 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3be79990e8425591742312f888e83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:25.952796 17142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3be79990e8425591742312f888e83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:25.952896 17276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3be79990e8425591742312f888e83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:25.953487 17589 ts_tablet_manager.cc:1918] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:15:25.953645 17589 log.cc:1196] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/a3c57019d2fe44678774329c48ee7d22
I20260924 03:15:25.954147 17589 ts_tablet_manager.cc:1939] T a3c57019d2fe44678774329c48ee7d22 P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting consensus metadata
I20260924 03:15:25.956107 16987 catalog_manager.cc:4841] TS b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953): tablet a3c57019d2fe44678774329c48ee7d22 (table test-table [id=baf466dd62ff40d58558fc5ca76ccaae]) successfully deleted
I20260924 03:15:25.958983 17600 tablet_bootstrap.cc:492] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: Bootstrap starting.
I20260924 03:15:25.959328 17601 tablet_bootstrap.cc:492] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: Bootstrap starting.
I20260924 03:15:25.959358 17599 tablet_bootstrap.cc:492] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: Bootstrap starting.
I20260924 03:15:25.960631 17600 tablet_bootstrap.cc:654] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:25.960904 17599 tablet_bootstrap.cc:654] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:25.961320 17601 tablet_bootstrap.cc:654] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:25.970628 17601 tablet_bootstrap.cc:492] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: No bootstrap required, opened a new log
I20260924 03:15:25.970808 17601 ts_tablet_manager.cc:1397] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260924 03:15:25.970909 17600 tablet_bootstrap.cc:492] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: No bootstrap required, opened a new log
I20260924 03:15:25.971071 17600 ts_tablet_manager.cc:1397] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260924 03:15:25.971566 17601 raft_consensus.cc:348] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.971827 17601 raft_consensus.cc:374] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:25.971907 17601 raft_consensus.cc:729] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450df1629adc458ba9d3cd8f3f6008fb, State: Initialized, Role: FOLLOWER
I20260924 03:15:25.971815 17600 raft_consensus.cc:348] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.972106 17600 raft_consensus.cc:374] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:25.972204 17600 raft_consensus.cc:729] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e233f82279040c382b829f3ec3edc92, State: Initialized, Role: FOLLOWER
I20260924 03:15:25.972154 17601 consensus_queue.cc:260] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.972440 17600 consensus_queue.cc:260] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.972959 17600 ts_tablet_manager.cc:1428] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 03:15:25.974385 17601 ts_tablet_manager.cc:1428] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 03:15:25.975446 17599 tablet_bootstrap.cc:492] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: No bootstrap required, opened a new log
I20260924 03:15:25.975639 17599 ts_tablet_manager.cc:1397] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260924 03:15:25.976418 17599 raft_consensus.cc:348] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.976701 17599 raft_consensus.cc:374] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:25.976794 17599 raft_consensus.cc:729] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17cb4c30ad8455e89135d44a5ea6c0b, State: Initialized, Role: FOLLOWER
I20260924 03:15:25.977027 17599 consensus_queue.cc:260] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:25.977517 17599 ts_tablet_manager.cc:1428] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:15:26.288132 17605 raft_consensus.cc:482] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:26.288434 17605 raft_consensus.cc:504] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:26.289340 17605 leader_election.cc:288] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783), b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953)
I20260924 03:15:26.289932 17295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3be79990e8425591742312f888e83a" candidate_uuid: "450df1629adc458ba9d3cd8f3f6008fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" is_pre_election: true
I20260924 03:15:26.289909 17161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3be79990e8425591742312f888e83a" candidate_uuid: "450df1629adc458ba9d3cd8f3f6008fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e233f82279040c382b829f3ec3edc92" is_pre_election: true
I20260924 03:15:26.290279 17161 raft_consensus.cc:2454] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450df1629adc458ba9d3cd8f3f6008fb in term 0.
I20260924 03:15:26.290262 17295 raft_consensus.cc:2454] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450df1629adc458ba9d3cd8f3f6008fb in term 0.
I20260924 03:15:26.290740 17369 leader_election.cc:302] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450df1629adc458ba9d3cd8f3f6008fb, b17cb4c30ad8455e89135d44a5ea6c0b; no voters:
I20260924 03:15:26.291006 17605 raft_consensus.cc:2781] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:26.291141 17605 raft_consensus.cc:482] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:26.291230 17605 raft_consensus.cc:3037] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:26.293394 17605 raft_consensus.cc:504] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:26.293922 17605 leader_election.cc:288] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [CANDIDATE]: Term 1 election: Requested vote from peers 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783), b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953)
I20260924 03:15:26.294417 17161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3be79990e8425591742312f888e83a" candidate_uuid: "450df1629adc458ba9d3cd8f3f6008fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e233f82279040c382b829f3ec3edc92"
I20260924 03:15:26.294626 17161 raft_consensus.cc:3037] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:26.294910 17295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3be79990e8425591742312f888e83a" candidate_uuid: "450df1629adc458ba9d3cd8f3f6008fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b"
I20260924 03:15:26.295125 17295 raft_consensus.cc:3037] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:26.297137 17295 raft_consensus.cc:2454] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450df1629adc458ba9d3cd8f3f6008fb in term 1.
I20260924 03:15:26.297544 17369 leader_election.cc:302] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450df1629adc458ba9d3cd8f3f6008fb, b17cb4c30ad8455e89135d44a5ea6c0b; no voters:
I20260924 03:15:26.297785 17605 raft_consensus.cc:2781] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:26.297874 17161 raft_consensus.cc:2454] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450df1629adc458ba9d3cd8f3f6008fb in term 1.
I20260924 03:15:26.302187 17605 raft_consensus.cc:686] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 1 LEADER]: Becoming Leader. State: Replica: 450df1629adc458ba9d3cd8f3f6008fb, State: Running, Role: LEADER
I20260924 03:15:26.302668 17605 consensus_queue.cc:237] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } }
I20260924 03:15:26.305307 17000 catalog_manager.cc:5495] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131). New cstate: current_term: 1 leader_uuid: "450df1629adc458ba9d3cd8f3f6008fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "450df1629adc458ba9d3cd8f3f6008fb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 41433 } health_report { overall_health: HEALTHY } } }
I20260924 03:15:26.338099 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:26.333096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.14.227.131:41433)
0924 03:15:26.333654 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:26.333658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:26.333674 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:15:26.333718 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:26.333721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:26.333776 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:26.333907 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:26.334319 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:26.335042 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:26.336357 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:26.336776 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:26.336905 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:26.337310 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:26.337410 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:26.337506 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:26.337658 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 03:15:26.337726 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":292,"threads_started":1}
I20260924 03:15:26.347419 17161 raft_consensus.cc:1261] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Refusing update from remote peer 450df1629adc458ba9d3cd8f3f6008fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:26.347995 17295 raft_consensus.cc:1261] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Refusing update from remote peer 450df1629adc458ba9d3cd8f3f6008fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:26.348066 17619 consensus_queue.cc:1035] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e233f82279040c382b829f3ec3edc92" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:26.348503 17605 consensus_queue.cc:1035] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17cb4c30ad8455e89135d44a5ea6c0b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:15:26.371318 17571 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 03:15:27.952083 17000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:27.952342 17000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:27.955960 17000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7e1f0d6e0314ee4934f77ccb4b1fcd4: Sending DeleteTablet for 3 replicas of tablet 8e3be79990e8425591742312f888e83a
I20260924 03:15:27.956681 17142 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3be79990e8425591742312f888e83a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:27 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56398
I20260924 03:15:27.956739 17276 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3be79990e8425591742312f888e83a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:27 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:57734
I20260924 03:15:27.956818 17415 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3be79990e8425591742312f888e83a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:27 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38604
I20260924 03:15:27.957384 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17071
I20260924 03:15:27.969187 17649 tablet_replica.cc:330] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: stopping tablet replica
I20260924 03:15:27.969547 17649 raft_consensus.cc:2229] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:27.969964 17649 raft_consensus.cc:2258] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:27.970582 17648 tablet_replica.cc:330] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: stopping tablet replica
I20260924 03:15:27.970810 17647 tablet_replica.cc:330] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: stopping tablet replica
I20260924 03:15:27.970813 17648 raft_consensus.cc:2229] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:27.971024 17648 raft_consensus.cc:2258] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:27.971117 17649 ts_tablet_manager.cc:1905] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:27.971151 17647 raft_consensus.cc:2229] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:15:27.972302 17648 ts_tablet_manager.cc:1905] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:27.972357 17647 raft_consensus.cc:2258] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:27.973529 17647 ts_tablet_manager.cc:1905] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:27.978211 17649 ts_tablet_manager.cc:1918] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:15:27.978413 17649 log.cc:1196] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/8e3be79990e8425591742312f888e83a
I20260924 03:15:27.979107 17648 ts_tablet_manager.cc:1918] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:15:27.979265 17648 log.cc:1196] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/8e3be79990e8425591742312f888e83a
I20260924 03:15:27.979301 17649 ts_tablet_manager.cc:1939] T 8e3be79990e8425591742312f888e83a P 6e233f82279040c382b829f3ec3edc92: Deleting consensus metadata
I20260924 03:15:27.979748 17648 ts_tablet_manager.cc:1939] T 8e3be79990e8425591742312f888e83a P b17cb4c30ad8455e89135d44a5ea6c0b: Deleting consensus metadata
I20260924 03:15:27.979794 17647 ts_tablet_manager.cc:1918] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 03:15:27.979952 17647 log.cc:1196] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/8e3be79990e8425591742312f888e83a
I20260924 03:15:27.980397 16984 catalog_manager.cc:4841] TS 6e233f82279040c382b829f3ec3edc92 (127.14.227.129:39783): tablet 8e3be79990e8425591742312f888e83a (table test-table [id=2daf91aa6b6348bf9e604dfa3886ab12]) successfully deleted
I20260924 03:15:27.980759 17647 ts_tablet_manager.cc:1939] T 8e3be79990e8425591742312f888e83a P 450df1629adc458ba9d3cd8f3f6008fb: Deleting consensus metadata
I20260924 03:15:27.980821 16987 catalog_manager.cc:4841] TS b17cb4c30ad8455e89135d44a5ea6c0b (127.14.227.130:45953): tablet 8e3be79990e8425591742312f888e83a (table test-table [id=2daf91aa6b6348bf9e604dfa3886ab12]) successfully deleted
Sep 24 03:15:27 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:27.982128 16987 catalog_manager.cc:4841] TS 450df1629adc458ba9d3cd8f3f6008fb (127.14.227.131:41433): tablet 8e3be79990e8425591742312f888e83a (table test-table [id=2daf91aa6b6348bf9e604dfa3886ab12]) successfully deleted
I20260924 03:15:27.988251 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:27.958427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.14.227.129:39783)
0924 03:15:27.971749 (+ 13322us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:27.971755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:27.971776 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:15:27.971859 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:27.971865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:27.971939 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:27.972063 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:27.973911 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:27.975105 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:27.976980 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:27.977677 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:27.977763 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:27.982953 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:27.982970 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:27.982990 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:27.983048 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:27.986092 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:27.986631 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:27.986637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:27.986639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:27.986765 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:27.987089 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:27.987092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:27.987094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:27.987395 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:27.987596 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:27.987841 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 03:15:27.987990 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13179,"thread_start_us":13076,"threads_started":1}
W20260924 03:15:27.988940 17202 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56428
W20260924 03:15:27.989485 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17071: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:27.989666 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 17071
I20260924 03:15:28.015201 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17211
Sep 24 03:15:28 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:15:28.029834 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:28.015869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.14.227.130:45953)
0924 03:15:28.016413 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:28.016417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:28.016433 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:15:28.016485 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:28.016489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:28.016545 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:28.016637 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:28.017834 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:28.018621 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:28.019822 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:28.020178 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:28.020271 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:28.024160 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:28.024172 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:28.024187 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:28.024220 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:28.027109 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:28.027817 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:28.027824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:28.027827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:28.027955 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:28.028328 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:28.028333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:28.028335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:28.028695 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:28.028984 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:28.029277 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 03:15:28.029496 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":326,"threads_started":1}
W20260924 03:15:28.030510 17341 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57754
W20260924 03:15:28.031087 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17211: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:28.031239 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 17211
I20260924 03:15:28.052062 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17350
Sep 24 03:15:28 dist-test-slave-vd0p krb5kdc[16955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:15:28.066272 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:28.052786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.14.227.131:41433)
0924 03:15:28.053248 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:28.053253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:28.053269 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:15:28.053318 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:28.053322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:28.053384 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:28.053471 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:28.054760 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:28.055481 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:28.056585 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:28.056979 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:28.057074 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:28.061027 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:28.061053 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:28.061067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:28.061095 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:28.063928 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:28.064579 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:28.064586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:28.064588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:28.064696 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:28.065014 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:28.065017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:28.065019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:28.065345 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:28.065547 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:28.065740 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 03:15:28.065884 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":221,"threads_started":1}
W20260924 03:15:28.066924 17480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38644
W20260924 03:15:28.067523 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17350: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:28.067735 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 17350
I20260924 03:15:28.088924 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 16968
I20260924 03:15:28.090090 17040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:28.204108 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 16968
2026-09-24T03:15:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8335 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:28 dist-test-slave-vd0p krb5kdc[17657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:28 dist-test-slave-vd0p krb5kdc[17657](info): set up 2 sockets
Sep 24 03:15:28 dist-test-slave-vd0p krb5kdc[17657](info): commencing operation
krb5kdc: starting...
W20260924 03:15:29.290051 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:29 dist-test-slave-vd0p krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:29.470072 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33649
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:33649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:29.779146 17670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:29.779409 17670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:29.779583 17670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:29.786631 17670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:29.786753 17670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:29.786852 17670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:29.786943 17670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:29.787024 17670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:29.796489 17670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:33649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:29.797761 17670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:29.799113 17670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:29.810551 17676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:29.810575 17678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:29.810655 17675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:29.811254 17670 server_base.cc:1034] running on GCE node
I20260924 03:15:29.811990 17670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:29.813330 17670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:29.814533 17670 hybrid_clock.cc:648] HybridClock initialized: now 1790219729814491 us; error 40 us; skew 500 ppm
I20260924 03:15:29.814913 17670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:29 dist-test-slave-vd0p krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219729, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:29.818100 17670 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:29.819576 17670 webserver.cc:460] Webserver started at http://127.14.227.190:33837/ using document root <none> and password file <none>
I20260924 03:15:29.820251 17670 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:29.820384 17670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:29.820688 17670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:29.823589 17670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "1683c543fd8a4f0698cb6f2373be6f36"
format_stamp: "Formatted at 2026-09-24 03:15:29 on dist-test-slave-vd0p"
server_key: "f0a26d2bf653219e3b8999e15c502883"
server_key_iv: "dbec833b6c97faf7aed9e82dc485b638"
server_key_version: "encryptionkey@0"
I20260924 03:15:29.824296 17670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "1683c543fd8a4f0698cb6f2373be6f36"
format_stamp: "Formatted at 2026-09-24 03:15:29 on dist-test-slave-vd0p"
server_key: "f0a26d2bf653219e3b8999e15c502883"
server_key_iv: "dbec833b6c97faf7aed9e82dc485b638"
server_key_version: "encryptionkey@0"
I20260924 03:15:29.829124 17670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 03:15:29.832479 17685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:29.833812 17670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:29.834038 17670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "1683c543fd8a4f0698cb6f2373be6f36"
format_stamp: "Formatted at 2026-09-24 03:15:29 on dist-test-slave-vd0p"
server_key: "f0a26d2bf653219e3b8999e15c502883"
server_key_iv: "dbec833b6c97faf7aed9e82dc485b638"
server_key_version: "encryptionkey@0"
I20260924 03:15:29.834219 17670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:29.845996 17670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:29.849594 17670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:29.849859 17670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:29.867377 17670 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:33649
I20260924 03:15:29.867399 17736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:33649 every 8 connection(s)
I20260924 03:15:29.868830 17670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:29.872219 17737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:29.875808 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 17670
I20260924 03:15:29.876044 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:29.876847 15246 external_mini_cluster.cc:1428] Setting key da884701dc790bb411a3b3cb767a02a9
I20260924 03:15:29.885008 17737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36: Bootstrap starting.
I20260924 03:15:29.888605 17737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:29.889885 17737 log.cc:824] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:15:29 dist-test-slave-vd0p krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:29.892650 17737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36: No bootstrap required, opened a new log
I20260924 03:15:29.898919 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:29.878774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.14.227.190:33649)
0924 03:15:29.879522 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:29.879533 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:29.879571 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:15:29.881151 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:29.881171 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:29.881558 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:29.881860 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:29.883522 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:29.884759 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:29.886513 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:29.887135 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:29.887255 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:29.891938 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:29.891945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:29.891958 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:29.891993 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:29.895005 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:29.895652 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:29.895658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:29.895664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:29.895822 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:29.896202 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:29.896211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:29.896213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:29.896707 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:29.897049 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:29.897497 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 03:15:29.897879 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":357,"threads_started":1}
I20260924 03:15:29.900518 17737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1683c543fd8a4f0698cb6f2373be6f36" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33649 } }
I20260924 03:15:29.900854 17737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:29.900974 17737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1683c543fd8a4f0698cb6f2373be6f36, State: Initialized, Role: FOLLOWER
I20260924 03:15:29.901587 17737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1683c543fd8a4f0698cb6f2373be6f36" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33649 } }
I20260924 03:15:29.901877 17737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:29.902030 17737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:29.902208 17737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:29.904224 17737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1683c543fd8a4f0698cb6f2373be6f36" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33649 } }
I20260924 03:15:29.904692 17737 leader_election.cc:302] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1683c543fd8a4f0698cb6f2373be6f36; no voters:
I20260924 03:15:29.905243 17737 leader_election.cc:288] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:29.905352 17742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:29.906776 17742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [term 1 LEADER]: Becoming Leader. State: Replica: 1683c543fd8a4f0698cb6f2373be6f36, State: Running, Role: LEADER
I20260924 03:15:29.907401 17737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:29.907336 17742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1683c543fd8a4f0698cb6f2373be6f36" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33649 } }
I20260924 03:15:29.909978 17744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1683c543fd8a4f0698cb6f2373be6f36. Latest consensus state: current_term: 1 leader_uuid: "1683c543fd8a4f0698cb6f2373be6f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1683c543fd8a4f0698cb6f2373be6f36" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33649 } } }
I20260924 03:15:29.910261 17743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1683c543fd8a4f0698cb6f2373be6f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1683c543fd8a4f0698cb6f2373be6f36" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33649 } } }
I20260924 03:15:29.910543 17743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:29.910511 17744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:29.922660 17749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:29.928184 17749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:29.937794 17749 catalog_manager.cc:1347] Generated new cluster ID: ec19a81f7e6d434dbb7c5150849d52f2
I20260924 03:15:29.937908 17749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:29.970562 17749 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:29.971932 17749 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:29.978470 17749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1683c543fd8a4f0698cb6f2373be6f36: Generated new TSK 0
I20260924 03:15:29.979399 17749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:30.038080 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33649
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:30.355016 17764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:30.355304 17764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:30.355464 17764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:30.362496 17764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:30.362630 17764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:30.362808 17764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:30.372339 17764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:30.373514 17764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:30.374915 17764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:30.387844 17770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:30.388013 17769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:30.387956 17772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:30.388466 17764 server_base.cc:1034] running on GCE node
I20260924 03:15:30.389066 17764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:30.389912 17764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:30.391160 17764 hybrid_clock.cc:648] HybridClock initialized: now 1790219730391100 us; error 56 us; skew 500 ppm
I20260924 03:15:30.391448 17764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:30 dist-test-slave-vd0p krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219730, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:30.394539 17764 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:30.395960 17764 webserver.cc:460] Webserver started at http://127.14.227.129:36349/ using document root <none> and password file <none>
I20260924 03:15:30.396603 17764 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:30.396746 17764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:30.397049 17764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:30.399844 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "3a70b6bba1a24af98efdb3535ac3d391"
format_stamp: "Formatted at 2026-09-24 03:15:30 on dist-test-slave-vd0p"
server_key: "d3676548b764c1f977bfde9fca138763"
server_key_iv: "025b11dd7965ed57c81a5465725591c9"
server_key_version: "encryptionkey@0"
I20260924 03:15:30.400599 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "3a70b6bba1a24af98efdb3535ac3d391"
format_stamp: "Formatted at 2026-09-24 03:15:30 on dist-test-slave-vd0p"
server_key: "d3676548b764c1f977bfde9fca138763"
server_key_iv: "025b11dd7965ed57c81a5465725591c9"
server_key_version: "encryptionkey@0"
I20260924 03:15:30.405404 17764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 03:15:30.409126 17779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:30.410516 17764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:30.410672 17764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "3a70b6bba1a24af98efdb3535ac3d391"
format_stamp: "Formatted at 2026-09-24 03:15:30 on dist-test-slave-vd0p"
server_key: "d3676548b764c1f977bfde9fca138763"
server_key_iv: "025b11dd7965ed57c81a5465725591c9"
server_key_version: "encryptionkey@0"
I20260924 03:15:30.410868 17764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:30.428925 17764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:30.433135 17764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:30.433420 17764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:30.434377 17764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:30.435766 17764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:30.435866 17764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:30.436012 17764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:30.436105 17764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:30.469134 17764 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:37057
I20260924 03:15:30.469182 17891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:37057 every 8 connection(s)
I20260924 03:15:30.470500 17764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:30.476357 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 17764
I20260924 03:15:30.476581 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:30.477214 15246 external_mini_cluster.cc:1428] Setting key f94d4f629d4eebd35d95f4b5e039ad49
Sep 24 03:15:30 dist-test-slave-vd0p krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219730, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:30.490490 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:30.472913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:43445 (local address 127.14.227.190:33649)
0924 03:15:30.473356 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:30.473363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:30.474404 (+ 1041us) server_negotiation.cc:407] Connection header received
0924 03:15:30.476161 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:30.476166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:30.476259 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:30.476376 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:30.478042 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:30.479104 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:30.480591 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:30.481131 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:30.481263 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:30.484888 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:30.484893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:30.484896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:30.484945 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:30.487518 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:30.488231 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:30.488236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:30.488252 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:30.488363 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:30.488784 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:30.488789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:30.488793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:30.489044 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:30.489198 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:30.489875 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 03:15:30.490119 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":200,"threads_started":1}
I20260924 03:15:30.491416 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:30.473512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33649 (local address 127.14.227.129:43445)
0924 03:15:30.474256 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:30.474292 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:30.475880 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:30.476606 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:30.476615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:30.477143 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:30.477853 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:30.477871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:30.479262 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:30.479273 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:30.480447 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:30.480457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:30.481361 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:30.481368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:30.481595 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:30.482318 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:30.482344 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:30.484629 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:30.487658 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:30.487666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:30.487682 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:30.488103 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:30.488471 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:30.488477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:30.488480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:30.488666 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:30.489172 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:30.489179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:30.489514 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 03:15:30.489783 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 03:15:30.490326 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":273,"threads_started":1}
I20260924 03:15:30.493094 17892 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33649
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:15:30.493382 17892 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:30.494163 17892 heartbeater.cc:511] Master 127.14.227.190:33649 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:15:30.496500 17702 ts_manager.cc:194] Registered new tserver with Master: 3a70b6bba1a24af98efdb3535ac3d391 (127.14.227.129:37057)
I20260924 03:15:30.498697 17702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:43445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:30.519555 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33649
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:30.829681 17898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:30.830025 17898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:30.830192 17898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:30.837179 17898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:30.837321 17898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:30.837522 17898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:30.846886 17898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:30.848130 17898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:30.849400 17898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:30.862169 17904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:30.862252 17903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:30.862252 17906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:30.863767 17898 server_base.cc:1034] running on GCE node
I20260924 03:15:30.864355 17898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:30.865242 17898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:30.866456 17898 hybrid_clock.cc:648] HybridClock initialized: now 1790219730866420 us; error 49 us; skew 500 ppm
I20260924 03:15:30.866818 17898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:30 dist-test-slave-vd0p krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219730, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:30.870666 17898 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:30.872078 17898 webserver.cc:460] Webserver started at http://127.14.227.130:42699/ using document root <none> and password file <none>
I20260924 03:15:30.872745 17898 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:30.872891 17898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:30.873183 17898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:30.876006 17898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "c41576373217419c8639c5e369ce42b8"
format_stamp: "Formatted at 2026-09-24 03:15:30 on dist-test-slave-vd0p"
server_key: "2cf372549797a3e8eb18c0dbe466d783"
server_key_iv: "ab6543d288eb3d321c8c058c6a5f043f"
server_key_version: "encryptionkey@0"
I20260924 03:15:30.876760 17898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "c41576373217419c8639c5e369ce42b8"
format_stamp: "Formatted at 2026-09-24 03:15:30 on dist-test-slave-vd0p"
server_key: "2cf372549797a3e8eb18c0dbe466d783"
server_key_iv: "ab6543d288eb3d321c8c058c6a5f043f"
server_key_version: "encryptionkey@0"
I20260924 03:15:30.881829 17898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260924 03:15:30.885123 17913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:30.886497 17898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:30.886664 17898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "c41576373217419c8639c5e369ce42b8"
format_stamp: "Formatted at 2026-09-24 03:15:30 on dist-test-slave-vd0p"
server_key: "2cf372549797a3e8eb18c0dbe466d783"
server_key_iv: "ab6543d288eb3d321c8c058c6a5f043f"
server_key_version: "encryptionkey@0"
I20260924 03:15:30.886860 17898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:30.915544 17898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:30.920161 17898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:30.920460 17898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:30.921394 17898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:30.922875 17898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:30.922974 17898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:30.923123 17898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:30.923197 17898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:30.955065 17898 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:44373
I20260924 03:15:30.955113 18025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:44373 every 8 connection(s)
I20260924 03:15:30.956364 17898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:30.966019 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 17898
I20260924 03:15:30.966213 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:30.966790 15246 external_mini_cluster.cc:1428] Setting key 06d9587ebdbd89c2c132eaf1ce4cfda9
Sep 24 03:15:30 dist-test-slave-vd0p krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219730, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:30.975415 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:30.958849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:57419 (local address 127.14.227.190:33649)
0924 03:15:30.959027 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:30.959031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:30.960165 (+ 1134us) server_negotiation.cc:407] Connection header received
0924 03:15:30.961859 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:30.961863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:30.961927 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:30.962065 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:30.963486 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:30.964156 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:30.965650 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:30.966090 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:30.966233 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:30.969732 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:30.969736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:30.969739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:30.969767 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:30.972474 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:30.973193 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:30.973201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:30.973208 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:30.973323 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:30.973739 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:30.973746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:30.973751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:30.974060 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:30.974213 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:30.974857 (+ 644us) server_negotiation.cc:299] Negotiation successful
0924 03:15:30.975072 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:15:30.976318 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:30.959347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33649 (local address 127.14.227.130:57419)
0924 03:15:30.960006 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:30.960043 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:30.961614 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:30.962262 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:30.962270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:30.962794 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:30.963344 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:30.963360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:30.964280 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:30.964283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:30.965522 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:30.965533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:30.966318 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:30.966325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:30.966615 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:30.967300 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:30.967346 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:30.969516 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:30.972610 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:30.972618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:30.972634 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:30.973034 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:30.973439 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:30.973444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:30.973447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:30.973621 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:30.974188 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:30.974195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:30.974508 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 03:15:30.974768 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 03:15:30.975265 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":255,"threads_started":1}
I20260924 03:15:30.977713 18026 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33649
I20260924 03:15:30.978063 18026 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:30.978724 18026 heartbeater.cc:511] Master 127.14.227.190:33649 requested a full tablet report, sending...
I20260924 03:15:30.980183 17702 ts_manager.cc:194] Registered new tserver with Master: c41576373217419c8639c5e369ce42b8 (127.14.227.130:44373)
I20260924 03:15:30.981443 17702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:57419
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:31.008838 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33649
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:31.322338 18031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:31.322630 18031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:31.322793 18031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:31.330271 18031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:31.330436 18031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:31.330621 18031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:31.340240 18031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:31.341557 18031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:31.343019 18031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:31.356279 18036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:31.356256 18039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:31.356343 18037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:31.357132 18031 server_base.cc:1034] running on GCE node
I20260924 03:15:31.357846 18031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:31.358842 18031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:31.360072 18031 hybrid_clock.cc:648] HybridClock initialized: now 1790219731360024 us; error 38 us; skew 500 ppm
I20260924 03:15:31.360344 18031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:31 dist-test-slave-vd0p krb5kdc[17657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219731, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:31.363360 18031 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:31.364841 18031 webserver.cc:460] Webserver started at http://127.14.227.131:44739/ using document root <none> and password file <none>
I20260924 03:15:31.365510 18031 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:31.365656 18031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:31.366074 18031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:31.368881 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "4ec13beb28b545a6aeb2bc904231887a"
format_stamp: "Formatted at 2026-09-24 03:15:31 on dist-test-slave-vd0p"
server_key: "137d55f49dc5da000d881ffae12f78dc"
server_key_iv: "49ff7c3aa277a85f246f4d7cd43adb5b"
server_key_version: "encryptionkey@0"
I20260924 03:15:31.369555 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "4ec13beb28b545a6aeb2bc904231887a"
format_stamp: "Formatted at 2026-09-24 03:15:31 on dist-test-slave-vd0p"
server_key: "137d55f49dc5da000d881ffae12f78dc"
server_key_iv: "49ff7c3aa277a85f246f4d7cd43adb5b"
server_key_version: "encryptionkey@0"
I20260924 03:15:31.374446 18031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:15:31.377533 18046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:31.378804 18031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:31.378980 18031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "4ec13beb28b545a6aeb2bc904231887a"
format_stamp: "Formatted at 2026-09-24 03:15:31 on dist-test-slave-vd0p"
server_key: "137d55f49dc5da000d881ffae12f78dc"
server_key_iv: "49ff7c3aa277a85f246f4d7cd43adb5b"
server_key_version: "encryptionkey@0"
I20260924 03:15:31.379163 18031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:31.403889 18031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:31.408257 18031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:31.408545 18031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:31.409494 18031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:31.410880 18031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:31.410979 18031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:31.411126 18031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:31.411195 18031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:31.443523 18031 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:36517
I20260924 03:15:31.443578 18158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:36517 every 8 connection(s)
I20260924 03:15:31.444790 18031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:31.445395 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 18031
I20260924 03:15:31.445588 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:31.446079 15246 external_mini_cluster.cc:1428] Setting key 39577fdeb7eff02a27a235d0cb0552f6
Sep 24 03:15:31 dist-test-slave-vd0p krb5kdc[17657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219731, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:31.464787 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:31.447868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:43493 (local address 127.14.227.190:33649)
0924 03:15:31.448048 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:31.448054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:31.449410 (+ 1356us) server_negotiation.cc:407] Connection header received
0924 03:15:31.451177 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:31.451181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:31.451269 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:31.451346 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:31.452708 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:31.453448 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:31.454948 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:31.455332 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:31.455424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:31.458596 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:31.458600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:31.458603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:31.458632 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:31.462081 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:31.462691 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:31.462695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:31.462698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:31.462805 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:31.463134 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:31.463137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:31.463138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:31.463382 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:31.463483 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:31.464079 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 03:15:31.464225 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:15:31.465413 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:31.448384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33649 (local address 127.14.227.131:43493)
0924 03:15:31.449260 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:31.449308 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:31.450919 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:31.451538 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:31.451545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:31.452035 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:31.452565 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:31.452580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:31.453574 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:31.453578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:31.454829 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:31.454840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:31.455501 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:31.455505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:31.455681 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:31.456266 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:31.456286 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:31.458422 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:31.462186 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:31.462192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:31.462203 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:31.462583 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:31.462885 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:31.462888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:31.462890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:31.463047 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:31.463479 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:31.463484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:31.463788 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 03:15:31.463998 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 03:15:31.464371 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":367,"threads_started":1}
I20260924 03:15:31.466701 18159 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33649
I20260924 03:15:31.466992 18159 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:31.467674 18159 heartbeater.cc:511] Master 127.14.227.190:33649 requested a full tablet report, sending...
I20260924 03:15:31.469007 17702 ts_manager.cc:194] Registered new tserver with Master: 4ec13beb28b545a6aeb2bc904231887a (127.14.227.131:36517)
I20260924 03:15:31.469745 17702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:43493
I20260924 03:15:31.473688 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:31.502396 17892 heartbeater.cc:503] Master 127.14.227.190:33649 was elected leader, sending a full tablet report...
W20260924 03:15:31.504129 17894 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:31.501464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.14.227.190:33649)
0924 03:15:31.501635 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:31.501640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:31.502181 (+ 541us) server_negotiation.cc:407] Connection header received
0924 03:15:31.502346 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:31.502352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:31.502431 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:31.502556 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:31.503861 (+ 1305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56304: BlockingRecv error: recv got EOF from 127.0.0.1:56304 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:15:31.507182 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17764
W20260924 03:15:31.509884 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17764: Not authorized: Client connection negotiation failed: client connection to 127.14.227.129:37057: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:15:31.510097 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 17764
I20260924 03:15:31.525055 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17898
W20260924 03:15:31.527506 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17898: Not authorized: Client connection negotiation failed: client connection to 127.14.227.130:44373: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:15:31.527655 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 17898
I20260924 03:15:31.542426 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18031
W20260924 03:15:31.545027 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18031: Not authorized: Client connection negotiation failed: client connection to 127.14.227.131:36517: server requires authentication, but client does not have Kerberos credentials available
I20260924 03:15:31.545176 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 18031
I20260924 03:15:31.559788 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 17670
I20260924 03:15:31.560849 17732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:31.662395 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 17670
2026-09-24T03:15:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3455 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:31 dist-test-slave-vd0p krb5kdc[18176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:31 dist-test-slave-vd0p krb5kdc[18176](info): set up 2 sockets
Sep 24 03:15:31 dist-test-slave-vd0p krb5kdc[18176](info): commencing operation
krb5kdc: starting...
W20260924 03:15:32.730263 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:32 dist-test-slave-vd0p krb5kdc[18176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:32.910032 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46027
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:46027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:33.218467 18189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:33.218730 18189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:33.218817 18189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:33.225864 18189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:33.226024 18189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:33.226099 18189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:33.226154 18189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:33.226202 18189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:33.235991 18189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:46027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:33.237167 18189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:33.238382 18189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:33.249766 18194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:33.249825 18197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:33.249828 18195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:33.251029 18189 server_base.cc:1034] running on GCE node
I20260924 03:15:33.251747 18189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:33.253051 18189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:33.254263 18189 hybrid_clock.cc:648] HybridClock initialized: now 1790219733254217 us; error 31 us; skew 500 ppm
I20260924 03:15:33.254639 18189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:33 dist-test-slave-vd0p krb5kdc[18176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219733, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:33.257839 18189 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:33.259311 18189 webserver.cc:460] Webserver started at http://127.14.227.190:39107/ using document root <none> and password file <none>
I20260924 03:15:33.259968 18189 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:33.260076 18189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:33.260367 18189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:33.263397 18189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "98701254e11f414984abb5b7be8038a8"
format_stamp: "Formatted at 2026-09-24 03:15:33 on dist-test-slave-vd0p"
server_key: "179d1c15385e8c9e29a5064210aca790"
server_key_iv: "8433abd564875befa81d8797cb1b8e7c"
server_key_version: "encryptionkey@0"
I20260924 03:15:33.264180 18189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "98701254e11f414984abb5b7be8038a8"
format_stamp: "Formatted at 2026-09-24 03:15:33 on dist-test-slave-vd0p"
server_key: "179d1c15385e8c9e29a5064210aca790"
server_key_iv: "8433abd564875befa81d8797cb1b8e7c"
server_key_version: "encryptionkey@0"
I20260924 03:15:33.269381 18189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:15:33.272782 18204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:33.274159 18189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:15:33.274338 18189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "98701254e11f414984abb5b7be8038a8"
format_stamp: "Formatted at 2026-09-24 03:15:33 on dist-test-slave-vd0p"
server_key: "179d1c15385e8c9e29a5064210aca790"
server_key_iv: "8433abd564875befa81d8797cb1b8e7c"
server_key_version: "encryptionkey@0"
I20260924 03:15:33.274598 18189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:33.294412 18189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:33.298472 18189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:33.298772 18189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:33.317691 18189 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:46027
I20260924 03:15:33.317714 18255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:46027 every 8 connection(s)
I20260924 03:15:33.319108 18189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:33.322691 18256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:33.325590 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 18189
I20260924 03:15:33.325809 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:33.326467 15246 external_mini_cluster.cc:1428] Setting key 3db7363f1274a6b4038f2c683a868dba
I20260924 03:15:33.335196 18256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8: Bootstrap starting.
I20260924 03:15:33.338603 18256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:33.339540 18256 log.cc:824] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:15:33 dist-test-slave-vd0p krb5kdc[18176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:33.342440 18256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8: No bootstrap required, opened a new log
I20260924 03:15:33.349488 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:33.328174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.14.227.190:46027)
0924 03:15:33.329007 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:33.329021 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:33.329070 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:15:33.331055 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:33.331084 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:33.331592 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:33.332035 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:33.333324 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:33.335248 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:33.336445 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:33.337497 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:33.337665 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:33.341341 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:33.341351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:33.341367 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:33.341405 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:33.345200 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:33.345810 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:33.345818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:33.345826 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:33.346069 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:33.346409 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:33.346415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:33.346418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:33.347105 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:33.347391 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:33.347711 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 03:15:33.348139 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":368,"threads_started":1}
I20260924 03:15:33.352792 18256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98701254e11f414984abb5b7be8038a8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46027 } }
I20260924 03:15:33.353224 18256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:33.353333 18256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98701254e11f414984abb5b7be8038a8, State: Initialized, Role: FOLLOWER
I20260924 03:15:33.353439 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:15:33.353924 18256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98701254e11f414984abb5b7be8038a8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46027 } }
I20260924 03:15:33.354187 18256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:33.354312 18256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:33.354483 18256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:33.356492 18256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98701254e11f414984abb5b7be8038a8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46027 } }
I20260924 03:15:33.356951 18256 leader_election.cc:302] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98701254e11f414984abb5b7be8038a8; no voters:
I20260924 03:15:33.357519 18256 leader_election.cc:288] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:33.357654 18261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:33.359498 18256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:33.360217 18261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [term 1 LEADER]: Becoming Leader. State: Replica: 98701254e11f414984abb5b7be8038a8, State: Running, Role: LEADER
I20260924 03:15:33.360746 18261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98701254e11f414984abb5b7be8038a8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46027 } }
I20260924 03:15:33.366190 18263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98701254e11f414984abb5b7be8038a8. Latest consensus state: current_term: 1 leader_uuid: "98701254e11f414984abb5b7be8038a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98701254e11f414984abb5b7be8038a8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46027 } } }
I20260924 03:15:33.366556 18263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:33.367070 18262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98701254e11f414984abb5b7be8038a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98701254e11f414984abb5b7be8038a8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46027 } } }
I20260924 03:15:33.367329 18262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:33.378880 18273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:33.383965 18273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:33.394672 18273 catalog_manager.cc:1347] Generated new cluster ID: bb96c2698a724b649a4e13928a05f3ac
I20260924 03:15:33.394832 18273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:33.406677 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:33.399166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.14.227.190:46027)
0924 03:15:33.399360 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:33.399369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:33.399935 (+ 566us) server_negotiation.cc:407] Connection header received
0924 03:15:33.400167 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:33.400173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:33.400252 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:33.400358 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:33.401348 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:33.402233 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:33.403556 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:33.404026 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:33.404157 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:33.405231 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:33.405236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:33.405239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:33.405818 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:33.405881 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:33.405908 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:33.406070 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:33.406205 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 03:15:33.406296 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:15:33.420790 18273 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:33.422129 18273 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:33.429819 18273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98701254e11f414984abb5b7be8038a8: Generated new TSK 0
I20260924 03:15:33.430893 18273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:15:33.440079 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18189
I20260924 03:15:33.441412 18251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:33.537408 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 18189
2026-09-24T03:15:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1870 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:33 dist-test-slave-vd0p krb5kdc[18291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:33 dist-test-slave-vd0p krb5kdc[18291](info): set up 2 sockets
Sep 24 03:15:33 dist-test-slave-vd0p krb5kdc[18291](info): commencing operation
krb5kdc: starting...
W20260924 03:15:34.614488 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.005s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:34 dist-test-slave-vd0p krb5kdc[18291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:34.798056 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44605
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:44605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:35.111682 18304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:35.112021 18304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:35.112188 18304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:35.119372 18304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:35.119513 18304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:35.119620 18304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:35.119724 18304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:35.119805 18304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:35.129351 18304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:44605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:35.130659 18304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:35.131915 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:35.143121 18312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:35.143141 18310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:35.143440 18309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:35.144094 18304 server_base.cc:1034] running on GCE node
I20260924 03:15:35.144881 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:35.146258 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:35.147475 18304 hybrid_clock.cc:648] HybridClock initialized: now 1790219735147424 us; error 62 us; skew 500 ppm
I20260924 03:15:35.147821 18304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:35 dist-test-slave-vd0p krb5kdc[18291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219735, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:35.150833 18304 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:35.152180 18304 webserver.cc:460] Webserver started at http://127.14.227.190:46369/ using document root <none> and password file <none>
I20260924 03:15:35.152822 18304 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:35.152968 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:35.153255 18304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:35.156070 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "d203812cef0746bd9f8091db5924d871"
format_stamp: "Formatted at 2026-09-24 03:15:35 on dist-test-slave-vd0p"
server_key: "49e3643a8a156fda2c78777e71cd612c"
server_key_iv: "f6e962b3dff84f5b5351e9dad4ffd8ab"
server_key_version: "encryptionkey@0"
I20260924 03:15:35.156811 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "d203812cef0746bd9f8091db5924d871"
format_stamp: "Formatted at 2026-09-24 03:15:35 on dist-test-slave-vd0p"
server_key: "49e3643a8a156fda2c78777e71cd612c"
server_key_iv: "f6e962b3dff84f5b5351e9dad4ffd8ab"
server_key_version: "encryptionkey@0"
I20260924 03:15:35.161592 18304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:15:35.164875 18319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:35.166381 18304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:15:35.166599 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "d203812cef0746bd9f8091db5924d871"
format_stamp: "Formatted at 2026-09-24 03:15:35 on dist-test-slave-vd0p"
server_key: "49e3643a8a156fda2c78777e71cd612c"
server_key_iv: "f6e962b3dff84f5b5351e9dad4ffd8ab"
server_key_version: "encryptionkey@0"
I20260924 03:15:35.166831 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:35.179399 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:35.183758 18304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:35.184090 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:35.202751 18304 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:44605
I20260924 03:15:35.202776 18370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:44605 every 8 connection(s)
I20260924 03:15:35.204252 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:35.208000 18371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:35.214294 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 18304
I20260924 03:15:35.214519 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:35.215137 15246 external_mini_cluster.cc:1428] Setting key 63c94e10a03f45f006525d545be74b06
I20260924 03:15:35.219575 18371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871: Bootstrap starting.
I20260924 03:15:35.223043 18371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:35.224529 18371 log.cc:824] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:35.227981 18371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871: No bootstrap required, opened a new log
Sep 24 03:15:35 dist-test-slave-vd0p krb5kdc[18291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:35.239921 18371 raft_consensus.cc:348] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d203812cef0746bd9f8091db5924d871" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44605 } }
I20260924 03:15:35.240427 18371 raft_consensus.cc:374] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:35.240557 18371 raft_consensus.cc:729] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d203812cef0746bd9f8091db5924d871, State: Initialized, Role: FOLLOWER
I20260924 03:15:35.240868 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:35.217232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.14.227.190:44605)
0924 03:15:35.218075 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:35.218087 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:35.218127 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:15:35.219694 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:35.219751 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:35.220149 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:35.220511 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:35.221898 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:35.224021 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:35.226051 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:35.226718 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:35.226860 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:35.231705 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:35.231721 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:35.231741 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:35.231785 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:35.236129 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:35.236786 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:35.236795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:35.236804 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:35.237026 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:35.237375 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:35.237381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:35.237385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:35.238216 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:35.238551 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:35.238880 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 03:15:35.239338 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":371,"threads_started":1}
I20260924 03:15:35.241341 18371 consensus_queue.cc:260] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d203812cef0746bd9f8091db5924d871" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44605 } }
I20260924 03:15:35.241614 18371 raft_consensus.cc:388] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:35.241781 18371 raft_consensus.cc:482] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:35.242017 18371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:35.244509 18371 raft_consensus.cc:504] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d203812cef0746bd9f8091db5924d871" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44605 } }
I20260924 03:15:35.245121 18371 leader_election.cc:302] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d203812cef0746bd9f8091db5924d871; no voters:
I20260924 03:15:35.245831 18371 leader_election.cc:288] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:35.245982 18376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:35.246711 18376 raft_consensus.cc:686] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [term 1 LEADER]: Becoming Leader. State: Replica: d203812cef0746bd9f8091db5924d871, State: Running, Role: LEADER
I20260924 03:15:35.247260 18376 consensus_queue.cc:237] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d203812cef0746bd9f8091db5924d871" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44605 } }
I20260924 03:15:35.248046 18371 sys_catalog.cc:564] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:35.250206 18378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d203812cef0746bd9f8091db5924d871. Latest consensus state: current_term: 1 leader_uuid: "d203812cef0746bd9f8091db5924d871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d203812cef0746bd9f8091db5924d871" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44605 } } }
I20260924 03:15:35.250229 18377 sys_catalog.cc:455] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d203812cef0746bd9f8091db5924d871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d203812cef0746bd9f8091db5924d871" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44605 } } }
I20260924 03:15:35.250625 18378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:35.250633 18377 sys_catalog.cc:458] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:35.253154 18382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:35.258323 18382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:35.266791 18382 catalog_manager.cc:1347] Generated new cluster ID: 31633a7e532d4512b0623cc88456f0b0
I20260924 03:15:35.266901 18382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:35.278613 18382 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:35.280364 18382 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:35.286906 18382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d203812cef0746bd9f8091db5924d871: Generated new TSK 0
I20260924 03:15:35.287807 18382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:35.341084 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44605
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:35.664502 18397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:35.664772 18397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:35.664865 18397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:35.672199 18397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:35.672300 18397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:35.672469 18397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:35.682204 18397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:35.683451 18397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:35.684693 18397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:35.698172 18403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:35.698161 18405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:35.698254 18402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:35.699275 18397 server_base.cc:1034] running on GCE node
I20260924 03:15:35.699922 18397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:35.700800 18397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:35.702030 18397 hybrid_clock.cc:648] HybridClock initialized: now 1790219735701967 us; error 50 us; skew 500 ppm
I20260924 03:15:35.702306 18397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:35 dist-test-slave-vd0p krb5kdc[18291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219735, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:35.705365 18397 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:35.706873 18397 webserver.cc:460] Webserver started at http://127.14.227.129:46287/ using document root <none> and password file <none>
I20260924 03:15:35.707527 18397 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:35.707667 18397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:35.707995 18397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:35.710736 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "4cc70bbc06ad4fb1ac95bc2acdf40267"
format_stamp: "Formatted at 2026-09-24 03:15:35 on dist-test-slave-vd0p"
server_key: "1caff7a2aee7c8f3a3bc29ae3dd26304"
server_key_iv: "1a3b3a589927bb2d00cf86d09c4c6dc9"
server_key_version: "encryptionkey@0"
I20260924 03:15:35.711469 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "4cc70bbc06ad4fb1ac95bc2acdf40267"
format_stamp: "Formatted at 2026-09-24 03:15:35 on dist-test-slave-vd0p"
server_key: "1caff7a2aee7c8f3a3bc29ae3dd26304"
server_key_iv: "1a3b3a589927bb2d00cf86d09c4c6dc9"
server_key_version: "encryptionkey@0"
I20260924 03:15:35.716220 18397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:15:35.719470 18412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:35.720685 18397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:15:35.720856 18397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "4cc70bbc06ad4fb1ac95bc2acdf40267"
format_stamp: "Formatted at 2026-09-24 03:15:35 on dist-test-slave-vd0p"
server_key: "1caff7a2aee7c8f3a3bc29ae3dd26304"
server_key_iv: "1a3b3a589927bb2d00cf86d09c4c6dc9"
server_key_version: "encryptionkey@0"
I20260924 03:15:35.721076 18397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:35.744500 18397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:35.748618 18397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:35.748947 18397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:35.749886 18397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:35.751209 18397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:35.751307 18397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:35.751457 18397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:35.751528 18397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:35.783886 18397 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:36459
I20260924 03:15:35.783972 18524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:36459 every 8 connection(s)
I20260924 03:15:35.785195 18397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:35.788571 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 18397
I20260924 03:15:35.788771 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:35.789494 15246 external_mini_cluster.cc:1428] Setting key 3685dd8884cde2d98996038417f8492e
Sep 24 03:15:35 dist-test-slave-vd0p krb5kdc[18291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219735, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:35.807652 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:35.787885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:45013 (local address 127.14.227.190:44605)
0924 03:15:35.788441 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:35.788448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:35.789688 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 03:15:35.791781 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:35.791786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:35.791877 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:35.791996 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:35.793512 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:35.794672 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:35.796353 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:35.796751 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:35.796841 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:35.800837 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:35.800845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:35.800852 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:35.800907 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:35.804485 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:35.805159 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:35.805168 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:35.805174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:35.805330 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:35.805730 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:35.805738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:35.805743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:35.806202 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:35.806355 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:35.807012 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 03:15:35.807246 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":275,"threads_started":1}
I20260924 03:15:35.808323 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:35.788562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44605 (local address 127.14.227.129:45013)
0924 03:15:35.789500 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:35.789555 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:35.791520 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:35.792271 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:35.792279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:35.792772 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:35.793344 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:35.793362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:35.794825 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:35.794829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:35.796189 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:35.796206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:35.796983 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:35.796992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:35.797311 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:35.798046 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:35.798079 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:35.800620 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:35.804613 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:35.804619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:35.804631 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:35.805036 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:35.805449 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:35.805452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:35.805454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:35.805619 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:35.806322 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:35.806328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:35.806621 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 03:15:35.806859 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 03:15:35.807234 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":452,"threads_started":1}
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:15:35.809857 18525 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44605
I20260924 03:15:35.810194 18525 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:35.810912 18525 heartbeater.cc:511] Master 127.14.227.190:44605 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:15:35.813311 18336 ts_manager.cc:194] Registered new tserver with Master: 4cc70bbc06ad4fb1ac95bc2acdf40267 (127.14.227.129:36459)
I20260924 03:15:35.815507 18336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:45013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:35.833283 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44605
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:36.145139 18531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:36.145399 18531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:36.145491 18531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:36.152650 18531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:36.152752 18531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:36.152863 18531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:36.162348 18531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:36.163532 18531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:36.164734 18531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:36.177650 18536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:36.177574 18537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:36.177933 18539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:36.178907 18531 server_base.cc:1034] running on GCE node
I20260924 03:15:36.179638 18531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:36.180629 18531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:36.181870 18531 hybrid_clock.cc:648] HybridClock initialized: now 1790219736181826 us; error 54 us; skew 500 ppm
I20260924 03:15:36.182214 18531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:36.185523 18531 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:36.187027 18531 webserver.cc:460] Webserver started at http://127.14.227.130:45449/ using document root <none> and password file <none>
I20260924 03:15:36.187722 18531 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:36.187884 18531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:36.188187 18531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:36.191277 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "1c2a2af5214048138ad312d2dd84cdb5"
format_stamp: "Formatted at 2026-09-24 03:15:36 on dist-test-slave-vd0p"
server_key: "72f596d645e52e77f9699af846d87f78"
server_key_iv: "2965f6a5c7129830a80df6db17198bdc"
server_key_version: "encryptionkey@0"
I20260924 03:15:36.192050 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "1c2a2af5214048138ad312d2dd84cdb5"
format_stamp: "Formatted at 2026-09-24 03:15:36 on dist-test-slave-vd0p"
server_key: "72f596d645e52e77f9699af846d87f78"
server_key_iv: "2965f6a5c7129830a80df6db17198bdc"
server_key_version: "encryptionkey@0"
I20260924 03:15:36.197036 18531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:15:36.200400 18546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:36.201730 18531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:36.201913 18531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "1c2a2af5214048138ad312d2dd84cdb5"
format_stamp: "Formatted at 2026-09-24 03:15:36 on dist-test-slave-vd0p"
server_key: "72f596d645e52e77f9699af846d87f78"
server_key_iv: "2965f6a5c7129830a80df6db17198bdc"
server_key_version: "encryptionkey@0"
I20260924 03:15:36.202199 18531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:36.223501 18531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:36.228468 18531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:36.228852 18531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:36.230005 18531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:36.231428 18531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:36.231511 18531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:36.231616 18531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:36.231666 18531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:36.263626 18531 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:41833
I20260924 03:15:36.263679 18658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:41833 every 8 connection(s)
I20260924 03:15:36.264907 18531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:36.269290 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 18531
I20260924 03:15:36.269477 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:36.270038 15246 external_mini_cluster.cc:1428] Setting key 58dfbcfc6fcf045dd343b0d26cf25552
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:36.287374 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:36.267525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:47723 (local address 127.14.227.190:44605)
0924 03:15:36.267716 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:36.267720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:36.268869 (+ 1149us) server_negotiation.cc:407] Connection header received
0924 03:15:36.271140 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:36.271148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:36.271237 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:36.271353 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:36.272998 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.274280 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.275874 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.276598 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.276734 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:36.280170 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:36.280179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:36.280186 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:36.280233 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:36.283995 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.284702 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.284710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.284717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.284875 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.285367 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.285373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.285377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.285733 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:36.285890 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:36.286602 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 03:15:36.286882 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:15:36.288173 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:36.267995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44605 (local address 127.14.227.130:47723)
0924 03:15:36.268700 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:36.268743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:36.270805 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:36.271584 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:36.271593 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:36.272228 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:36.272832 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:36.272850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.274457 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.274461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:36.275726 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:36.275738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.276803 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.276807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:36.277002 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:36.277730 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:36.277756 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:36.279934 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:36.284132 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:36.284139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:36.284152 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:36.284583 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:36.285011 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:36.285016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:36.285019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:36.285254 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:36.285861 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:36.285867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:36.286242 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 03:15:36.286473 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 03:15:36.286876 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":272,"threads_started":1}
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
I20260924 03:15:36.289726 18659 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44605
I20260924 03:15:36.290096 18659 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:36.290819 18659 heartbeater.cc:511] Master 127.14.227.190:44605 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
I20260924 03:15:36.292303 18336 ts_manager.cc:194] Registered new tserver with Master: 1c2a2af5214048138ad312d2dd84cdb5 (127.14.227.130:41833)
I20260924 03:15:36.293557 18336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:47723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:36.316862 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44605
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:36.631907 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:36.632192 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:36.632387 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:36.639550 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:36.639657 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:36.639870 18664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:36.650430 18664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:36.651705 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:36.653178 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:36.666451 18670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:36.666443 18672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:36.666483 18669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:36.667196 18664 server_base.cc:1034] running on GCE node
I20260924 03:15:36.667809 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:36.668730 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:36.670029 18664 hybrid_clock.cc:648] HybridClock initialized: now 1790219736669918 us; error 66 us; skew 500 ppm
I20260924 03:15:36.670395 18664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:36.673686 18664 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:36.675122 18664 webserver.cc:460] Webserver started at http://127.14.227.131:37251/ using document root <none> and password file <none>
I20260924 03:15:36.675789 18664 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:36.675935 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:36.676237 18664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:36.679131 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "84bd344fe6094c20ba9f873a68579166"
format_stamp: "Formatted at 2026-09-24 03:15:36 on dist-test-slave-vd0p"
server_key: "9299632c53f484d8d400d65a23cdc381"
server_key_iv: "1fe914d3f9f83d0c6143215cd34edc42"
server_key_version: "encryptionkey@0"
I20260924 03:15:36.679888 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "84bd344fe6094c20ba9f873a68579166"
format_stamp: "Formatted at 2026-09-24 03:15:36 on dist-test-slave-vd0p"
server_key: "9299632c53f484d8d400d65a23cdc381"
server_key_iv: "1fe914d3f9f83d0c6143215cd34edc42"
server_key_version: "encryptionkey@0"
I20260924 03:15:36.684950 18664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 03:15:36.688254 18679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:36.689616 18664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:36.689783 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "84bd344fe6094c20ba9f873a68579166"
format_stamp: "Formatted at 2026-09-24 03:15:36 on dist-test-slave-vd0p"
server_key: "9299632c53f484d8d400d65a23cdc381"
server_key_iv: "1fe914d3f9f83d0c6143215cd34edc42"
server_key_version: "encryptionkey@0"
I20260924 03:15:36.689996 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:36.707108 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:36.710917 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:36.711198 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:36.712132 18664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:36.713461 18664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:36.713558 18664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:36.713708 18664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:36.713778 18664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:36.746112 18664 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:38841
I20260924 03:15:36.746153 18791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:38841 every 8 connection(s)
I20260924 03:15:36.747426 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:36.754315 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 18664
I20260924 03:15:36.754482 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:36.754957 15246 external_mini_cluster.cc:1428] Setting key b8b3490679deaef2fe2afc7009e7e9ab
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:36.767764 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:36.749911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:41921 (local address 127.14.227.190:44605)
0924 03:15:36.750150 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:36.750157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:36.751566 (+ 1409us) server_negotiation.cc:407] Connection header received
0924 03:15:36.753400 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:36.753404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:36.753496 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:36.753591 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:36.755483 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.756498 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.758457 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.758870 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.759001 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:36.762364 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:36.762387 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:36.762390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:36.762438 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:36.764756 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.765461 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.765467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.765472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.765586 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.766068 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.766073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.766076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.766338 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:36.766488 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:36.767201 (+ 713us) server_negotiation.cc:299] Negotiation successful
0924 03:15:36.767414 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 03:15:36.768748 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:36.750522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44605 (local address 127.14.227.131:41921)
0924 03:15:36.751407 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:36.751445 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:36.753134 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:36.753994 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:36.754006 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:36.754667 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:36.755312 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:36.755330 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.756648 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.756654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:36.758305 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:36.758316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.759107 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.759115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:36.759360 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:36.759948 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:36.759973 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:36.762160 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:36.764907 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:36.764915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:36.764931 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:36.765332 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:36.765701 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:36.765706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:36.765709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:36.765902 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:36.766490 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:36.766498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:36.766845 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 03:15:36.767118 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 03:15:36.767624 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":453,"threads_started":1}
I20260924 03:15:36.770256 18792 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44605
I20260924 03:15:36.770563 18792 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:36.771395 18792 heartbeater.cc:511] Master 127.14.227.190:44605 requested a full tablet report, sending...
I20260924 03:15:36.772754 18336 ts_manager.cc:194] Registered new tserver with Master: 84bd344fe6094c20ba9f873a68579166 (127.14.227.131:38841)
I20260924 03:15:36.773602 18336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:41921
I20260924 03:15:36.782192 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260924 03:15:36.818445 18525 heartbeater.cc:503] Master 127.14.227.190:44605 was elected leader, sending a full tablet report...
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:36.828766 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:36.814516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.14.227.190:44605)
0924 03:15:36.814762 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:36.814771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:36.815112 (+ 341us) server_negotiation.cc:407] Connection header received
0924 03:15:36.815258 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:36.815262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:36.815329 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:36.815416 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:36.816741 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.817577 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.819028 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.819439 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.819542 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:36.823498 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:36.823503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:36.823507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:36.823539 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:36.826210 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.826854 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.826859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.826861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.826986 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.827364 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.827370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.827373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.827685 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:36.827926 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:36.828228 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 03:15:36.828436 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 03:15:36.829314 18336 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45784
I20260924 03:15:36.831131 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18397
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:36.846522 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:36.832103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.14.227.129:36459)
0924 03:15:36.832706 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:36.832713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:36.832737 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:15:36.832793 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:36.832799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:36.833005 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:36.833138 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:36.834623 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.835675 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.836799 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.837337 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.837428 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:36.841171 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:36.841181 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:36.841194 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:36.841223 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:36.843818 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.844505 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.844520 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.844524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.844641 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.844997 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.845003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.845008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.845326 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:36.845615 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:36.845894 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 03:15:36.846186 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":304,"threads_started":1}
W20260924 03:15:36.847772 18520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41136
W20260924 03:15:36.848875 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18397: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:36.849100 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 18397
I20260924 03:15:36.864048 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18531
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:15:36.879261 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:36.864861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.14.227.130:41833)
0924 03:15:36.865445 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:36.865452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:36.865476 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:15:36.865539 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:36.865545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:36.865739 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:36.865865 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:36.867017 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.867989 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.869129 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.869649 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.869745 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:36.873794 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:36.873818 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:36.873834 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:36.873879 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:36.876750 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.877338 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.877345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.877347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.877482 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.877783 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.877786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.877788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.878268 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:36.878525 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:36.878729 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:15:36.878951 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":316,"threads_started":1}
W20260924 03:15:36.880553 18654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59686
W20260924 03:15:36.881670 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18531: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:36.881794 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 18531
I20260924 03:15:36.896700 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18664
Sep 24 03:15:36 dist-test-slave-vd0p krb5kdc[18291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219736, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:15:36.911890 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:36.897577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.14.227.131:38841)
0924 03:15:36.898247 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:36.898254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:36.898280 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:15:36.898340 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:36.898346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:36.898685 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:36.898859 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:36.900038 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.901035 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.902258 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:36.902798 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:36.902892 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:36.906920 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:36.906931 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:36.906944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:36.906975 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:36.909667 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.910307 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.910314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.910316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.910432 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:36.910718 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:36.910721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:36.910723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:36.911021 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:36.911232 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:36.911425 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 03:15:36.911615 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":296,"threads_started":1}
W20260924 03:15:36.913092 18787 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54598
W20260924 03:15:36.914165 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:15:36.914333 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 18664
I20260924 03:15:36.928853 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18304
I20260924 03:15:36.930049 18366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:37.045591 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 18304
2026-09-24T03:15:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3512 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:37 dist-test-slave-vd0p krb5kdc[18809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:37 dist-test-slave-vd0p krb5kdc[18809](info): set up 2 sockets
Sep 24 03:15:37 dist-test-slave-vd0p krb5kdc[18809](info): commencing operation
krb5kdc: starting...
W20260924 03:15:38.126297 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:38 dist-test-slave-vd0p krb5kdc[18809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:38.313589 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43597
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:43597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:38.634558 18822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:38.634895 18822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:38.635234 18822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:38.642350 18822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:38.642441 18822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:38.642544 18822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:38.642632 18822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:38.642719 18822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:38.652176 18822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:43597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:38.653442 18822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:38.654762 18822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:38.666427 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:38.666414 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:38.666437 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:38.667313 18822 server_base.cc:1034] running on GCE node
I20260924 03:15:38.668126 18822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:38.669450 18822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:38.670698 18822 hybrid_clock.cc:648] HybridClock initialized: now 1790219738670640 us; error 64 us; skew 500 ppm
I20260924 03:15:38.671033 18822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:38 dist-test-slave-vd0p krb5kdc[18809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219738, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:38.680276 18822 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:38.681769 18822 webserver.cc:460] Webserver started at http://127.14.227.190:45623/ using document root <none> and password file <none>
I20260924 03:15:38.682467 18822 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:38.682565 18822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:38.682822 18822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:38.685616 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "7c9dff6fa7db4e00a6705efc5e713eb0"
format_stamp: "Formatted at 2026-09-24 03:15:38 on dist-test-slave-vd0p"
server_key: "ffaa73109520c3635c8c88193c33540c"
server_key_iv: "43d4647272cee142946c2fdd8bddbad1"
server_key_version: "encryptionkey@0"
I20260924 03:15:38.686439 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "7c9dff6fa7db4e00a6705efc5e713eb0"
format_stamp: "Formatted at 2026-09-24 03:15:38 on dist-test-slave-vd0p"
server_key: "ffaa73109520c3635c8c88193c33540c"
server_key_iv: "43d4647272cee142946c2fdd8bddbad1"
server_key_version: "encryptionkey@0"
I20260924 03:15:38.691615 18822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:15:38.695043 18837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:38.696386 18822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:15:38.696565 18822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "7c9dff6fa7db4e00a6705efc5e713eb0"
format_stamp: "Formatted at 2026-09-24 03:15:38 on dist-test-slave-vd0p"
server_key: "ffaa73109520c3635c8c88193c33540c"
server_key_iv: "43d4647272cee142946c2fdd8bddbad1"
server_key_version: "encryptionkey@0"
I20260924 03:15:38.696762 18822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:38.724660 18822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:38.728672 18822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:38.728979 18822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:38.747113 18822 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:43597
I20260924 03:15:38.747156 18888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:43597 every 8 connection(s)
I20260924 03:15:38.748430 18822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:38.751361 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 18822
I20260924 03:15:38.751567 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:38.752192 15246 external_mini_cluster.cc:1428] Setting key d580593abf0ae94976a6a23316197e26
I20260924 03:15:38.752197 18889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:38.766052 18889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0: Bootstrap starting.
Sep 24 03:15:38 dist-test-slave-vd0p krb5kdc[18809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:38.769111 18889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:38.770224 18889 log.cc:824] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:38.772796 18889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0: No bootstrap required, opened a new log
I20260924 03:15:38.774278 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:38.754091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.14.227.190:43597)
0924 03:15:38.754846 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:38.754858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:38.754901 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:15:38.756559 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:38.756579 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:38.756963 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:38.757267 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:38.759020 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:38.760198 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:38.762004 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:38.762669 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:38.762819 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:38.767367 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:38.767378 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:38.767391 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:38.767428 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:38.770557 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:38.771221 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:38.771227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:38.771233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:38.771390 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:38.771775 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:38.771780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:38.771789 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:38.772247 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:38.772545 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:38.772921 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 03:15:38.773315 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":326,"threads_started":1}
I20260924 03:15:38.780727 18889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9dff6fa7db4e00a6705efc5e713eb0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43597 } }
I20260924 03:15:38.781055 18889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:38.781168 18889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9dff6fa7db4e00a6705efc5e713eb0, State: Initialized, Role: FOLLOWER
I20260924 03:15:38.781767 18889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9dff6fa7db4e00a6705efc5e713eb0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43597 } }
I20260924 03:15:38.782011 18889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:38.782156 18889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:38.782336 18889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:38.784600 18889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9dff6fa7db4e00a6705efc5e713eb0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43597 } }
I20260924 03:15:38.785071 18889 leader_election.cc:302] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c9dff6fa7db4e00a6705efc5e713eb0; no voters:
I20260924 03:15:38.785624 18889 leader_election.cc:288] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:38.785725 18894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:38.786510 18894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c9dff6fa7db4e00a6705efc5e713eb0, State: Running, Role: LEADER
I20260924 03:15:38.787052 18894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9dff6fa7db4e00a6705efc5e713eb0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43597 } }
I20260924 03:15:38.787608 18889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:38.789927 18895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c9dff6fa7db4e00a6705efc5e713eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9dff6fa7db4e00a6705efc5e713eb0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43597 } } }
I20260924 03:15:38.790405 18895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:38.790051 18896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c9dff6fa7db4e00a6705efc5e713eb0. Latest consensus state: current_term: 1 leader_uuid: "7c9dff6fa7db4e00a6705efc5e713eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9dff6fa7db4e00a6705efc5e713eb0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43597 } } }
I20260924 03:15:38.790822 18896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:38.792536 18903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:38.796449 18903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:38.804817 18903 catalog_manager.cc:1347] Generated new cluster ID: 44b82e7a87f54fb3a42fa2e674ecbcba
I20260924 03:15:38.804919 18903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:38.824179 18903 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:38.825981 18903 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:38.839259 18903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c9dff6fa7db4e00a6705efc5e713eb0: Generated new TSK 0
I20260924 03:15:38.840170 18903 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:38.894426 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43597
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:39.211432 18915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:39.211706 18915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:39.211862 18915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:39.218880 18915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:39.219014 18915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:39.219203 18915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:39.228986 18915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:39.230306 18915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:39.231712 18915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:39.245239 18920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:39.249619 18923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:39.249832 18921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:39.250296 18915 server_base.cc:1034] running on GCE node
I20260924 03:15:39.251000 18915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:39.252099 18915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:39.253343 18915 hybrid_clock.cc:648] HybridClock initialized: now 1790219739253293 us; error 39 us; skew 500 ppm
I20260924 03:15:39.253712 18915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:39 dist-test-slave-vd0p krb5kdc[18809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219739, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:39.257082 18915 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:39.258903 18915 webserver.cc:460] Webserver started at http://127.14.227.129:34729/ using document root <none> and password file <none>
I20260924 03:15:39.259660 18915 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:39.259799 18915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:39.260145 18915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:39.263314 18915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "ff7839295ccd410ca5c6d5a2f638a374"
format_stamp: "Formatted at 2026-09-24 03:15:39 on dist-test-slave-vd0p"
server_key: "3ac39e03bdbbe7f4e4e654fada1a872c"
server_key_iv: "9854a044a78b7cfb7f9535e801c539c8"
server_key_version: "encryptionkey@0"
I20260924 03:15:39.264129 18915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "ff7839295ccd410ca5c6d5a2f638a374"
format_stamp: "Formatted at 2026-09-24 03:15:39 on dist-test-slave-vd0p"
server_key: "3ac39e03bdbbe7f4e4e654fada1a872c"
server_key_iv: "9854a044a78b7cfb7f9535e801c539c8"
server_key_version: "encryptionkey@0"
I20260924 03:15:39.269232 18915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:15:39.272862 18930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:39.274385 18915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:15:39.274562 18915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "ff7839295ccd410ca5c6d5a2f638a374"
format_stamp: "Formatted at 2026-09-24 03:15:39 on dist-test-slave-vd0p"
server_key: "3ac39e03bdbbe7f4e4e654fada1a872c"
server_key_iv: "9854a044a78b7cfb7f9535e801c539c8"
server_key_version: "encryptionkey@0"
I20260924 03:15:39.274786 18915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:39.298355 18915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:39.302922 18915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:39.303242 18915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:39.304193 18915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:39.305526 18915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:39.305641 18915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:39.305800 18915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:39.305881 18915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:39.340287 18915 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:36003
I20260924 03:15:39.340344 19042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:36003 every 8 connection(s)
I20260924 03:15:39.341677 18915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:39.343277 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 18915
I20260924 03:15:39.343514 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:39.344308 15246 external_mini_cluster.cc:1428] Setting key 10e9b4299791cddececc7ed0f030ad06
Sep 24 03:15:39 dist-test-slave-vd0p krb5kdc[18809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219739, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:39.361507 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:39.344377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:39959 (local address 127.14.227.190:43597)
0924 03:15:39.345211 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:39.345217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:39.347931 (+ 2714us) server_negotiation.cc:407] Connection header received
0924 03:15:39.347981 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:39.347985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:39.348044 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:39.348161 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:39.349662 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:39.350450 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:39.351973 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:39.352383 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:39.352530 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:39.355973 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:39.355978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:39.355981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:39.356011 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:39.358616 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:39.359306 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:39.359312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:39.359315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:39.359445 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:39.359862 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:39.359867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:39.359870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:39.360133 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:39.360282 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:39.360939 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 03:15:39.361148 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":575,"threads_started":1}
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:15:39.362486 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:39.345061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43597 (local address 127.14.227.129:39959)
0924 03:15:39.345907 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:39.345978 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:39.347659 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:39.348406 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:39.348415 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:39.348940 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:39.349492 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:39.349509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:39.350616 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:39.350626 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:39.351826 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:39.351837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:39.352629 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:39.352636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:39.352874 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:39.353597 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:39.353619 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:39.355744 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:39.358758 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:39.358767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:39.358783 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:39.359188 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:39.359550 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:39.359556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:39.359560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:39.359748 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:39.360256 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:39.360264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:39.360573 (+ 309us) client_negotiation.cc:769] Sending connection context
0924 03:15:39.360835 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 03:15:39.361333 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":359,"threads_started":1}
I20260924 03:15:39.364096 19043 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43597
I20260924 03:15:39.364411 19043 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:15:39.365339 19043 heartbeater.cc:511] Master 127.14.227.190:43597 requested a full tablet report, sending...
I20260924 03:15:39.367576 18854 ts_manager.cc:194] Registered new tserver with Master: ff7839295ccd410ca5c6d5a2f638a374 (127.14.227.129:36003)
I20260924 03:15:39.368969 18854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:39959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:39.389611 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43597
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:39.703563 19049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:39.703825 19049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:39.703917 19049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:39.711086 19049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:39.711189 19049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:39.711306 19049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:39.720688 19049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:39.721863 19049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:39.723201 19049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:39.735792 19054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:39.735868 19055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:39.735829 19057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:39.736956 19049 server_base.cc:1034] running on GCE node
I20260924 03:15:39.737496 19049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:39.738401 19049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:39.739621 19049 hybrid_clock.cc:648] HybridClock initialized: now 1790219739739582 us; error 50 us; skew 500 ppm
I20260924 03:15:39.739910 19049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:39 dist-test-slave-vd0p krb5kdc[18809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219739, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:39.743111 19049 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:39.744542 19049 webserver.cc:460] Webserver started at http://127.14.227.130:33667/ using document root <none> and password file <none>
I20260924 03:15:39.745189 19049 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:39.745327 19049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:39.745626 19049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:39.749336 19049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "9038a6343a5f4228a43cb0a46edff4dc"
format_stamp: "Formatted at 2026-09-24 03:15:39 on dist-test-slave-vd0p"
server_key: "2b604fd675a30f487e3d28726545de01"
server_key_iv: "ab86351f43adcf8f3e9de06669548e0e"
server_key_version: "encryptionkey@0"
I20260924 03:15:39.750314 19049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "9038a6343a5f4228a43cb0a46edff4dc"
format_stamp: "Formatted at 2026-09-24 03:15:39 on dist-test-slave-vd0p"
server_key: "2b604fd675a30f487e3d28726545de01"
server_key_iv: "ab86351f43adcf8f3e9de06669548e0e"
server_key_version: "encryptionkey@0"
I20260924 03:15:39.756348 19049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 03:15:39.760293 19064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:39.761786 19049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:39.761951 19049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "9038a6343a5f4228a43cb0a46edff4dc"
format_stamp: "Formatted at 2026-09-24 03:15:39 on dist-test-slave-vd0p"
server_key: "2b604fd675a30f487e3d28726545de01"
server_key_iv: "ab86351f43adcf8f3e9de06669548e0e"
server_key_version: "encryptionkey@0"
I20260924 03:15:39.762185 19049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:39.787431 19049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:39.791927 19049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:39.792217 19049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:39.793213 19049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:39.794507 19049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:39.794610 19049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:39.794772 19049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:39.794847 19049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:39.826478 19049 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:43821
I20260924 03:15:39.826562 19176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:43821 every 8 connection(s)
I20260924 03:15:39.827775 19049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:39.829072 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 19049
I20260924 03:15:39.829303 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:39.829767 15246 external_mini_cluster.cc:1428] Setting key 014a65fc5f892562541702584f6ff42b
Sep 24 03:15:39 dist-test-slave-vd0p krb5kdc[18809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219739, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:39.848352 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:39.830851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:54937 (local address 127.14.227.190:43597)
0924 03:15:39.831049 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:39.831056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:39.832082 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 03:15:39.833967 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:39.833973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:39.834070 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:39.834178 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:39.835630 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:39.836762 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:39.838270 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:39.839026 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:39.839166 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:39.842533 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:39.842542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:39.842550 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:39.842595 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:39.845248 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:39.845920 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:39.845929 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:39.845935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:39.846130 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:39.846557 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:39.846566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:39.846570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:39.846928 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:39.847086 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:39.847713 (+ 627us) server_negotiation.cc:299] Negotiation successful
0924 03:15:39.847954 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:15:39.848970 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:39.831118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43597 (local address 127.14.227.130:54937)
0924 03:15:39.831934 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:39.831985 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:39.833687 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:39.834403 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:39.834412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:39.834914 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:39.835461 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:39.835477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:39.836927 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:39.836931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:39.838140 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:39.838151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:39.839213 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:39.839218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:39.839486 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:39.840205 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:39.840227 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:39.842313 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:39.845367 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:39.845373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:39.845384 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:39.845794 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:39.846271 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:39.846274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:39.846276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:39.846444 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:39.847046 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:39.847052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:39.847334 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 03:15:39.847572 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 03:15:39.847916 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
I20260924 03:15:39.850443 19177 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43597
I20260924 03:15:39.850741 19177 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:39.851485 19177 heartbeater.cc:511] Master 127.14.227.190:43597 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
I20260924 03:15:39.853044 18854 ts_manager.cc:194] Registered new tserver with Master: 9038a6343a5f4228a43cb0a46edff4dc (127.14.227.130:43821)
I20260924 03:15:39.853822 18854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:54937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:39.873282 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43597
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:40.183125 19182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:40.183379 19182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:40.183537 19182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:40.190632 19182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:40.190730 19182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:40.190891 19182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:40.200834 19182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:40.202198 19182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:40.203437 19182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:40.216144 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:40.216156 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:40.216169 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:40.217132 19182 server_base.cc:1034] running on GCE node
I20260924 03:15:40.217792 19182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:40.218794 19182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:40.220000 19182 hybrid_clock.cc:648] HybridClock initialized: now 1790219740219949 us; error 48 us; skew 500 ppm
I20260924 03:15:40.220309 19182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[18809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219740, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:40.223452 19182 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:40.224888 19182 webserver.cc:460] Webserver started at http://127.14.227.131:37915/ using document root <none> and password file <none>
I20260924 03:15:40.225518 19182 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:40.225615 19182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:40.225858 19182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:40.228652 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "6841e16aaa21494b8e114423420294da"
format_stamp: "Formatted at 2026-09-24 03:15:40 on dist-test-slave-vd0p"
server_key: "ad87d9744e5863d1d64dc90bc3f6490f"
server_key_iv: "cd83ef3a8664dd5a2715cd16c4f1b144"
server_key_version: "encryptionkey@0"
I20260924 03:15:40.229343 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "6841e16aaa21494b8e114423420294da"
format_stamp: "Formatted at 2026-09-24 03:15:40 on dist-test-slave-vd0p"
server_key: "ad87d9744e5863d1d64dc90bc3f6490f"
server_key_iv: "cd83ef3a8664dd5a2715cd16c4f1b144"
server_key_version: "encryptionkey@0"
I20260924 03:15:40.234323 19182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 03:15:40.237519 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:40.238812 19182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:40.238982 19182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "6841e16aaa21494b8e114423420294da"
format_stamp: "Formatted at 2026-09-24 03:15:40 on dist-test-slave-vd0p"
server_key: "ad87d9744e5863d1d64dc90bc3f6490f"
server_key_iv: "cd83ef3a8664dd5a2715cd16c4f1b144"
server_key_version: "encryptionkey@0"
I20260924 03:15:40.239167 19182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:40.252332 19182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:40.256601 19182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:40.256887 19182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:40.257746 19182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:40.258991 19182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:40.259059 19182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:40.259212 19182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:40.259259 19182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:40.292470 19182 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:45749
I20260924 03:15:40.292513 19309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:45749 every 8 connection(s)
I20260924 03:15:40.293697 19182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:40.299860 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 19182
I20260924 03:15:40.300094 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:40.300529 15246 external_mini_cluster.cc:1428] Setting key 87adf35e647249fbfc67e321e9dc6325
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[18809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219740, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:40.313926 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:40.296453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:34097 (local address 127.14.227.190:43597)
0924 03:15:40.296641 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:40.296646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:40.297878 (+ 1232us) server_negotiation.cc:407] Connection header received
0924 03:15:40.299789 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:40.299794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:40.299897 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:40.300003 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:40.301711 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.302701 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.304171 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.304587 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.304713 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:40.308075 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:40.308085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:40.308091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:40.308132 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:40.311094 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.311763 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.311769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.311773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.311906 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.312311 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.312319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.312322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.312587 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:40.312738 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:40.313392 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 03:15:40.313599 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:15:40.315114 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:40.297023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43597 (local address 127.14.227.131:34097)
0924 03:15:40.297709 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:40.297745 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:40.299496 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:40.300258 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:40.300266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:40.300858 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:40.301551 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:40.301579 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.302838 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.302843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:40.304029 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:40.304044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.304809 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.304815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:40.305047 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:40.305723 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:40.305745 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:40.307853 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:40.311226 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:40.311234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:40.311251 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:40.311646 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:40.312012 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:40.312018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:40.312022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:40.312201 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:40.312709 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:40.312717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:40.313036 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 03:15:40.313305 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 03:15:40.313779 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":272,"threads_started":1}
I20260924 03:15:40.316411 19310 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43597
I20260924 03:15:40.316717 19310 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:40.317379 19310 heartbeater.cc:511] Master 127.14.227.190:43597 requested a full tablet report, sending...
I20260924 03:15:40.318724 18854 ts_manager.cc:194] Registered new tserver with Master: 6841e16aaa21494b8e114423420294da (127.14.227.131:45749)
I20260924 03:15:40.319492 18854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:34097
I20260924 03:15:40.327813 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[18809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[18809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:40.372052 19043 heartbeater.cc:503] Master 127.14.227.190:43597 was elected leader, sending a full tablet report...
I20260924 03:15:40.374074 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:40.359150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.14.227.129:36003)
0924 03:15:40.359743 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:40.359754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:40.359784 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:15:40.359983 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:40.359991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:40.360215 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:40.360386 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:40.361802 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.362842 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.364080 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.364596 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.364759 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:40.368573 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:40.368583 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:40.368596 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:40.368626 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:40.371345 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.371906 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.371914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.371918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.372073 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.372454 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.372459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.372464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.372849 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:40.373139 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:40.373366 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 03:15:40.373635 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":266,"threads_started":1}
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[18809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:40.390818 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:40.377974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.14.227.190:43597)
0924 03:15:40.378158 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:40.378164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:40.378382 (+ 218us) server_negotiation.cc:407] Connection header received
0924 03:15:40.378604 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:40.378609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:40.378675 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:40.378773 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:40.380005 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.380780 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.381995 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.382381 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.382507 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:40.386167 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:40.386172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:40.386175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:40.386229 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:40.388614 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.389264 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.389268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.389270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.389388 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.389677 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.389680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.389682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.390021 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:40.390200 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:40.390400 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 03:15:40.390550 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 03:15:40.391409 18854 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35128
I20260924 03:15:40.392473 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18915
I20260924 03:15:40.404044 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:40.393373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.14.227.129:36003)
0924 03:15:40.393546 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:40.393550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:40.393798 (+ 248us) server_negotiation.cc:407] Connection header received
0924 03:15:40.393933 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:40.393937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:40.394079 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:40.394155 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:40.395194 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.395829 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.397022 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.397406 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.397508 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:40.399439 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:40.399444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:40.399447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:40.399481 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:40.402162 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.402724 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.402727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.402729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.402830 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.403106 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.403109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.403111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.403319 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:40.403475 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:40.403696 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 03:15:40.403837 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:15:40.404598 19038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:40.486685 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 18915
I20260924 03:15:40.502769 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 19049
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[18809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:15:40.517696 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:40.503686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.14.227.130:43821)
0924 03:15:40.504174 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:40.504182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:40.504205 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:15:40.504262 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:40.504268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:40.504506 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:40.504631 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:40.505780 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.506914 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.508127 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.508647 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.508739 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:40.512694 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:40.512704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:40.512717 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:40.512757 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:40.515386 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.515982 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.515989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.515991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.516103 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.516382 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.516385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.516387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.516699 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:40.516918 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:40.517146 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 03:15:40.517330 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":255,"threads_started":1}
I20260924 03:15:40.519225 19172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:40.601366 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 19049
I20260924 03:15:40.616765 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 19182
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[18809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:15:40.632601 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:40.617874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.14.227.131:45749)
0924 03:15:40.618529 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:40.618535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:40.618564 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:15:40.618623 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:40.618629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:40.618841 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:40.618965 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:40.620121 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.621137 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.622543 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:40.623097 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:40.623198 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:40.627282 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:40.627293 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:40.627307 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:40.627340 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:40.630201 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.630770 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.630777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.630779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.630891 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:40.631220 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:40.631222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:40.631224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:40.631548 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:40.631780 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:40.632033 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 03:15:40.632217 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":349,"threads_started":1}
I20260924 03:15:40.633818 19305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:40.715312 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 19182
I20260924 03:15:40.730960 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 18822
I20260924 03:15:40.732100 18884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:40.832945 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 18822
2026-09-24T03:15:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3787 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[19329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[19329](info): set up 2 sockets
Sep 24 03:15:40 dist-test-slave-vd0p krb5kdc[19329](info): commencing operation
krb5kdc: starting...
W20260924 03:15:41.938346 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:42 dist-test-slave-vd0p krb5kdc[19329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:42.121098 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46079
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:46079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:42.426441 19342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:42.426719 19342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:42.426884 19342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:42.433840 19342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:42.433939 19342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:42.434054 19342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:42.434156 19342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:42.434278 19342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:42.443957 19342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:46079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:42.445225 19342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:42.446620 19342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:42.458182 19347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:42.458266 19348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:42.458302 19350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:42.458891 19342 server_base.cc:1034] running on GCE node
I20260924 03:15:42.459664 19342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:42.460995 19342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:42.462231 19342 hybrid_clock.cc:648] HybridClock initialized: now 1790219742462192 us; error 51 us; skew 500 ppm
I20260924 03:15:42.462574 19342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:42 dist-test-slave-vd0p krb5kdc[19329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219742, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:42.465615 19342 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:42.466838 19342 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:42.466972 19342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:42.467293 19342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:42.470230 19342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "130d2116f4da4c5abf4e499d77e441d0"
format_stamp: "Formatted at 2026-09-24 03:15:42 on dist-test-slave-vd0p"
server_key: "80935a542c16b096f8d8d638cea14f4d"
server_key_iv: "81f185b5feed07608687dc1804b67990"
server_key_version: "encryptionkey@0"
I20260924 03:15:42.471017 19342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "130d2116f4da4c5abf4e499d77e441d0"
format_stamp: "Formatted at 2026-09-24 03:15:42 on dist-test-slave-vd0p"
server_key: "80935a542c16b096f8d8d638cea14f4d"
server_key_iv: "81f185b5feed07608687dc1804b67990"
server_key_version: "encryptionkey@0"
I20260924 03:15:42.475922 19342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:15:42.479135 19356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:42.480388 19342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:42.480575 19342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "130d2116f4da4c5abf4e499d77e441d0"
format_stamp: "Formatted at 2026-09-24 03:15:42 on dist-test-slave-vd0p"
server_key: "80935a542c16b096f8d8d638cea14f4d"
server_key_iv: "81f185b5feed07608687dc1804b67990"
server_key_version: "encryptionkey@0"
I20260924 03:15:42.480779 19342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:42.492151 19342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:42.495916 19342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:42.496179 19342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:42.513793 19342 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:46079
I20260924 03:15:42.513813 19407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:46079 every 8 connection(s)
I20260924 03:15:42.514645 19342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:42.517187 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 19342
I20260924 03:15:42.517409 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:42.518074 15246 external_mini_cluster.cc:1428] Setting key aab9707e063c9abcd2f2fc12e48b6567
I20260924 03:15:42.518304 19408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:42.532236 19408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0: Bootstrap starting.
Sep 24 03:15:42 dist-test-slave-vd0p krb5kdc[19329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:42.535369 19408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:42.536394 19408 log.cc:824] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:42.539011 19408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0: No bootstrap required, opened a new log
I20260924 03:15:42.541234 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:42.520004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.14.227.190:46079)
0924 03:15:42.520794 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:42.520804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:42.520845 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:15:42.522675 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:42.522694 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:42.523052 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:42.523402 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:42.525160 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:42.526496 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:42.528256 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:42.528945 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:42.529073 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:42.533814 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:42.533821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:42.533838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:42.533876 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:42.536897 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:42.537572 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:42.537584 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:42.537590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:42.537740 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:42.538191 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:42.538203 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:42.538206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:42.538701 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:42.538993 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:42.539494 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 03:15:42.539944 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":382,"threads_started":1}
I20260924 03:15:42.546947 19408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130d2116f4da4c5abf4e499d77e441d0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46079 } }
I20260924 03:15:42.547241 19408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:42.547315 19408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130d2116f4da4c5abf4e499d77e441d0, State: Initialized, Role: FOLLOWER
I20260924 03:15:42.547811 19408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130d2116f4da4c5abf4e499d77e441d0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46079 } }
I20260924 03:15:42.547993 19408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:42.548080 19408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:42.548214 19408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:42.550240 19408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130d2116f4da4c5abf4e499d77e441d0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46079 } }
I20260924 03:15:42.550663 19408 leader_election.cc:302] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 130d2116f4da4c5abf4e499d77e441d0; no voters:
I20260924 03:15:42.551138 19408 leader_election.cc:288] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:42.551294 19413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:42.552043 19413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [term 1 LEADER]: Becoming Leader. State: Replica: 130d2116f4da4c5abf4e499d77e441d0, State: Running, Role: LEADER
I20260924 03:15:42.552517 19413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130d2116f4da4c5abf4e499d77e441d0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46079 } }
I20260924 03:15:42.553395 19408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:42.555195 19414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "130d2116f4da4c5abf4e499d77e441d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130d2116f4da4c5abf4e499d77e441d0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46079 } } }
I20260924 03:15:42.555195 19415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 130d2116f4da4c5abf4e499d77e441d0. Latest consensus state: current_term: 1 leader_uuid: "130d2116f4da4c5abf4e499d77e441d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "130d2116f4da4c5abf4e499d77e441d0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46079 } } }
I20260924 03:15:42.555598 19414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:42.555692 19415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:42.557811 19418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:42.563361 19418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:42.571341 19418 catalog_manager.cc:1347] Generated new cluster ID: 7c269196c2ff48ac8bef4d73f0236c33
I20260924 03:15:42.571455 19418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:42.594547 19418 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:42.595923 19418 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:42.611613 19418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0: Generated new TSK 0
I20260924 03:15:42.612447 19418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:42.666381 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46079
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:42.980402 19434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:42.980661 19434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:42.980810 19434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:42.988703 19434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:42.988801 19434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:42.988999 19434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:42.998821 19434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:43.000105 19434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:43.001405 19434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:43.014349 19439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:43.014407 19440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:43.014429 19442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:43.015864 19434 server_base.cc:1034] running on GCE node
I20260924 03:15:43.016467 19434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:43.017441 19434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:43.018662 19434 hybrid_clock.cc:648] HybridClock initialized: now 1790219743018621 us; error 41 us; skew 500 ppm
I20260924 03:15:43.018980 19434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:43 dist-test-slave-vd0p krb5kdc[19329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219743, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:43.022023 19434 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:43.023229 19434 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:43.023360 19434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:43.023669 19434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:43.026547 19434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "c6c06458976447dea32b3e5c7f7cec59"
format_stamp: "Formatted at 2026-09-24 03:15:43 on dist-test-slave-vd0p"
server_key: "60d79d8c9f90f7f9eb1c9e49521cef61"
server_key_iv: "d669f9570d180ddcc80ac1a41bc2fe59"
server_key_version: "encryptionkey@0"
I20260924 03:15:43.027329 19434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "c6c06458976447dea32b3e5c7f7cec59"
format_stamp: "Formatted at 2026-09-24 03:15:43 on dist-test-slave-vd0p"
server_key: "60d79d8c9f90f7f9eb1c9e49521cef61"
server_key_iv: "d669f9570d180ddcc80ac1a41bc2fe59"
server_key_version: "encryptionkey@0"
I20260924 03:15:43.032132 19434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:15:43.035327 19448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:43.036580 19434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:15:43.036755 19434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "c6c06458976447dea32b3e5c7f7cec59"
format_stamp: "Formatted at 2026-09-24 03:15:43 on dist-test-slave-vd0p"
server_key: "60d79d8c9f90f7f9eb1c9e49521cef61"
server_key_iv: "d669f9570d180ddcc80ac1a41bc2fe59"
server_key_version: "encryptionkey@0"
I20260924 03:15:43.036978 19434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:43.081374 19434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:43.085583 19434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:43.085865 19434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:43.086826 19434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:43.088169 19434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:43.088304 19434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:43.088434 19434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:43.088522 19434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:43.120564 19434 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:39669
I20260924 03:15:43.120645 19560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:39669 every 8 connection(s)
I20260924 03:15:43.121393 19434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:43.124152 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 19434
I20260924 03:15:43.124372 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:43.124928 15246 external_mini_cluster.cc:1428] Setting key 4afdb7a6b5baddd3c136b4637836c54b
Sep 24 03:15:43 dist-test-slave-vd0p krb5kdc[19329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219743, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:43.141147 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:43.123793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:35511 (local address 127.14.227.190:46079)
0924 03:15:43.124262 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:43.124269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:43.125337 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 03:15:43.127467 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:43.127474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:43.127561 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:43.127674 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:43.129066 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:43.130224 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:43.131742 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:43.132132 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:43.132266 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:43.135601 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:43.135605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:43.135609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:43.135638 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:43.138137 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:43.138829 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:43.138837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:43.138842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:43.138960 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:43.139379 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:43.139387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:43.139390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:43.139696 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:43.139861 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:43.140542 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 03:15:43.140789 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260924 03:15:43.142117 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:43.124365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46079 (local address 127.14.227.129:35511)
0924 03:15:43.125161 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:43.125213 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:43.127191 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:43.127862 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:43.127870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:43.128377 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:43.128900 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:43.128917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:43.130369 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:43.130373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:43.131590 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:43.131603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:43.132347 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:43.132356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:43.132571 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:43.133242 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:43.133267 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:43.135386 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:43.138265 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:43.138273 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:43.138288 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:43.138710 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:43.139076 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:43.139080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:43.139084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:43.139269 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:43.139823 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:43.139831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:43.140189 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 03:15:43.140449 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 03:15:43.140944 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":346,"threads_started":1}
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:15:43.143666 19561 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46079
I20260924 03:15:43.143926 19561 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:43.144495 19561 heartbeater.cc:511] Master 127.14.227.190:46079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:15:43.146620 19373 ts_manager.cc:194] Registered new tserver with Master: c6c06458976447dea32b3e5c7f7cec59 (127.14.227.129:39669)
I20260924 03:15:43.147990 19373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:35511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:43.168705 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46079
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:43.480113 19567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:43.480381 19567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:43.480479 19567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:43.487546 19567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:43.487648 19567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:43.487766 19567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:43.497182 19567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:43.498409 19567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:43.499697 19567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:43.511790 19572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:43.511795 19573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:43.511891 19575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:43.513243 19567 server_base.cc:1034] running on GCE node
I20260924 03:15:43.513877 19567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:43.514964 19567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:43.516201 19567 hybrid_clock.cc:648] HybridClock initialized: now 1790219743516131 us; error 79 us; skew 500 ppm
I20260924 03:15:43.516531 19567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:43 dist-test-slave-vd0p krb5kdc[19329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219743, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:43.520082 19567 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:43.526309 19567 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:43.526425 19567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:43.526860 19567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:43.531255 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "be50e5ed65144913ad25501d4564d3e3"
format_stamp: "Formatted at 2026-09-24 03:15:43 on dist-test-slave-vd0p"
server_key: "ea80b6021e827e6f70382215ec041de5"
server_key_iv: "c06e054ae9c84385b70ddab3dfdc1337"
server_key_version: "encryptionkey@0"
I20260924 03:15:43.532356 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "be50e5ed65144913ad25501d4564d3e3"
format_stamp: "Formatted at 2026-09-24 03:15:43 on dist-test-slave-vd0p"
server_key: "ea80b6021e827e6f70382215ec041de5"
server_key_iv: "c06e054ae9c84385b70ddab3dfdc1337"
server_key_version: "encryptionkey@0"
I20260924 03:15:43.537722 19567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:15:43.541308 19581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:43.542575 19567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:43.542755 19567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "be50e5ed65144913ad25501d4564d3e3"
format_stamp: "Formatted at 2026-09-24 03:15:43 on dist-test-slave-vd0p"
server_key: "ea80b6021e827e6f70382215ec041de5"
server_key_iv: "c06e054ae9c84385b70ddab3dfdc1337"
server_key_version: "encryptionkey@0"
I20260924 03:15:43.542970 19567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:43.556360 19567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:43.560685 19567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:43.560969 19567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:43.561878 19567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:43.563189 19567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:43.563309 19567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:43.563445 19567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:43.563540 19567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:43.595686 19567 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:43437
I20260924 03:15:43.595688 19693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:43437 every 8 connection(s)
I20260924 03:15:43.596575 19567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:43.606097 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 19567
I20260924 03:15:43.606284 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:43.606702 15246 external_mini_cluster.cc:1428] Setting key c0aa9c2834a854455a12083fc62e37cf
Sep 24 03:15:43 dist-test-slave-vd0p krb5kdc[19329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219743, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:43.615896 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:43.599021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:44167 (local address 127.14.227.190:46079)
0924 03:15:43.599387 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:43.599391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:43.600445 (+ 1054us) server_negotiation.cc:407] Connection header received
0924 03:15:43.602221 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:43.602227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:43.602313 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:43.602427 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:43.603806 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:43.604537 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:43.606147 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:43.606649 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:43.606780 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:43.610507 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:43.610511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:43.610514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:43.610543 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:43.612979 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:43.613652 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:43.613658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:43.613661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:43.613777 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:43.614246 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:43.614251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:43.614254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:43.614507 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:43.614706 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:43.615344 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 03:15:43.615557 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:15:43.616849 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:43.599645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46079 (local address 127.14.227.130:44167)
0924 03:15:43.600298 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:43.600334 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:43.601907 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:43.602657 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:43.602665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:43.603130 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:43.603645 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:43.603662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:43.604690 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:43.604710 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:43.605988 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:43.606011 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:43.606870 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:43.606878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:43.607093 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:43.607864 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:43.607897 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:43.610298 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:43.613108 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:43.613116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:43.613132 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:43.613515 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:43.613888 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:43.613894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:43.613897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:43.614123 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:43.614668 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:43.614676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:43.615003 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 03:15:43.615260 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 03:15:43.615766 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":260,"threads_started":1}
I20260924 03:15:43.618330 19694 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46079
I20260924 03:15:43.618623 19694 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:43.619226 19694 heartbeater.cc:511] Master 127.14.227.190:46079 requested a full tablet report, sending...
I20260924 03:15:43.620605 19373 ts_manager.cc:194] Registered new tserver with Master: be50e5ed65144913ad25501d4564d3e3 (127.14.227.130:43437)
I20260924 03:15:43.621403 19373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:44167
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:43.649274 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46079
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:43.967319 19699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:43.967578 19699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:43.967674 19699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:43.974740 19699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:43.974856 19699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:43.975004 19699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:43.984650 19699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:43.986014 19699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:43.987215 19699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:44.002398 19705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:44.002480 19707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:44.002462 19704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:44.003173 19699 server_base.cc:1034] running on GCE node
I20260924 03:15:44.003726 19699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:44.004619 19699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:44.005841 19699 hybrid_clock.cc:648] HybridClock initialized: now 1790219744005794 us; error 46 us; skew 500 ppm
I20260924 03:15:44.006246 19699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:44 dist-test-slave-vd0p krb5kdc[19329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219744, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:44.020268 19699 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:44.021433 19699 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:44.021564 19699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:44.021906 19699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:44.024771 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "50567ad4ca3b40a5ab4866ce64b20164"
format_stamp: "Formatted at 2026-09-24 03:15:44 on dist-test-slave-vd0p"
server_key: "754f0e996f3d6615ce95e921be046c09"
server_key_iv: "bcf6cc1a9d15cfafd66821dc6df5548b"
server_key_version: "encryptionkey@0"
I20260924 03:15:44.025568 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "50567ad4ca3b40a5ab4866ce64b20164"
format_stamp: "Formatted at 2026-09-24 03:15:44 on dist-test-slave-vd0p"
server_key: "754f0e996f3d6615ce95e921be046c09"
server_key_iv: "bcf6cc1a9d15cfafd66821dc6df5548b"
server_key_version: "encryptionkey@0"
I20260924 03:15:44.030474 19699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 03:15:44.033851 19713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:44.035229 19699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:15:44.035401 19699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "50567ad4ca3b40a5ab4866ce64b20164"
format_stamp: "Formatted at 2026-09-24 03:15:44 on dist-test-slave-vd0p"
server_key: "754f0e996f3d6615ce95e921be046c09"
server_key_iv: "bcf6cc1a9d15cfafd66821dc6df5548b"
server_key_version: "encryptionkey@0"
I20260924 03:15:44.035629 19699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:44.048422 19699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:44.052398 19699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:44.052675 19699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:44.053709 19699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:44.055194 19699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:44.055308 19699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:44.055445 19699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:44.055541 19699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:44.089186 19699 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:39357
I20260924 03:15:44.089239 19825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:39357 every 8 connection(s)
I20260924 03:15:44.090132 19699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:44.098098 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 19699
I20260924 03:15:44.098313 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:44.098702 15246 external_mini_cluster.cc:1428] Setting key 5f6524b345174c3fe4bfc30b942e4623
Sep 24 03:15:44 dist-test-slave-vd0p krb5kdc[19329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219744, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:44.109237 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.092494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:54247 (local address 127.14.227.190:46079)
0924 03:15:44.092695 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.092702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.093903 (+ 1201us) server_negotiation.cc:407] Connection header received
0924 03:15:44.095645 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.095649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.095740 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.095826 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:44.097197 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.097900 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.099774 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.100264 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.100395 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.103858 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:44.103862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:44.103866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:44.103905 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:44.106371 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.107064 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.107071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.107074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.107196 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.107605 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.107610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.107613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.107870 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:44.108025 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.108679 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.108877 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:15:44.110224 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.093070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46079 (local address 127.14.227.131:54247)
0924 03:15:44.093737 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:44.093780 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:44.095401 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:44.096013 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:44.096023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:44.096502 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:44.097055 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.097074 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.098069 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.098077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.099606 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.099618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.100575 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.100584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.100808 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.101522 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:44.101566 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:44.103664 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:44.106504 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:44.106512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:44.106527 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:44.106938 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:44.107305 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:44.107311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:44.107315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:44.107481 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:44.107998 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:44.108005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:44.108335 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 03:15:44.108604 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 03:15:44.109089 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":255,"threads_started":1}
I20260924 03:15:44.111496 19826 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46079
I20260924 03:15:44.111809 19826 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:44.112510 19826 heartbeater.cc:511] Master 127.14.227.190:46079 requested a full tablet report, sending...
I20260924 03:15:44.114029 19373 ts_manager.cc:194] Registered new tserver with Master: 50567ad4ca3b40a5ab4866ce64b20164 (127.14.227.131:39357)
I20260924 03:15:44.114786 19373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:54247
I20260924 03:15:44.125669 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:44.142756 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.129967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.14.227.190:46079)
0924 03:15:44.130135 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.130140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.130653 (+ 513us) server_negotiation.cc:407] Connection header received
0924 03:15:44.130902 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.130906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.130965 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.131051 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:44.132363 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.133062 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.134444 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.134899 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.134998 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.137215 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:44.137220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:44.137223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:44.137253 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:44.139925 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.140788 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.140792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.140795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.140984 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.141401 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.141406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.141409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.141762 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:44.142064 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.142243 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.142401 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:15:44.150897 19561 heartbeater.cc:503] Master 127.14.227.190:46079 was elected leader, sending a full tablet report...
I20260924 03:15:44.146983 19373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:44.154304 19373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:44.190160 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.166615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39669 (local address 127.0.0.1:48008)
0924 03:15:44.179198 (+ 12583us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:44.179267 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:44.179487 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:44.182908 (+ 3421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:44.182917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:44.182962 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:44.183405 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.183419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.185305 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.185311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.187398 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.187409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.189071 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.189076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.189293 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.189346 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 03:15:44.189473 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:15:44.189675 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12240,"mutex_wait_us":1,"spinlock_wait_cycles":2816,"thread_start_us":13108,"threads_started":1}
I20260924 03:15:44.190915 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.166879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.14.227.129:39669)
0924 03:15:44.181866 (+ 14987us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.181877 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.181916 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:15:44.182066 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.182076 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.182392 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.182630 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:44.183654 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.185175 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.187546 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.188556 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.188758 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.188943 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.190387 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.190564 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14778,"thread_start_us":14588,"threads_started":1}
I20260924 03:15:44.191686 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.166711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.14.227.131:39357)
0924 03:15:44.181429 (+ 14718us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.181440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.182911 (+ 1471us) server_negotiation.cc:407] Connection header received
0924 03:15:44.183013 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.183021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.183279 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.183449 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:44.184203 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.185822 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.188448 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.189880 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.190104 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.191115 (+ 1011us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.191190 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.191365 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14535,"thread_start_us":14389,"threads_started":1}
I20260924 03:15:44.191891 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.182128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:43437 (local address 127.0.0.1:37246)
0924 03:15:44.182885 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:44.182906 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:44.183121 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:44.183979 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:44.183983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:44.184018 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:44.184401 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.184414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.186310 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.186316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.188505 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.188514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.190286 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.190295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.190464 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.190493 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:15:44.190587 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:15:44.190718 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":23,"spinlock_wait_cycles":6400,"thread_start_us":471,"threads_started":1}
I20260924 03:15:44.191905 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.180990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.14.227.130:43437)
0924 03:15:44.182099 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.182107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.182993 (+ 886us) server_negotiation.cc:407] Connection header received
0924 03:15:44.183320 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.183328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.183558 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.183707 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:44.184644 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.186152 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.188647 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.189789 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.189943 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.190140 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.191446 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.191613 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":835,"threads_started":1}
I20260924 03:15:44.191968 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.166655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:39357 (local address 127.0.0.1:43280)
0924 03:15:44.182625 (+ 15970us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:44.182650 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:44.182777 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:44.183678 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:44.183683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:44.183709 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:44.184013 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.184020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.186008 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.186014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.188282 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.188296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.190577 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.190583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.190751 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.190790 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:15:44.190872 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:15:44.191014 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15768,"mutex_wait_us":3,"thread_start_us":15594,"threads_started":1}
I20260924 03:15:44.194913 19496 tablet_service.cc:1467] Processing CreateTablet for tablet 27ed9a0b72a44ee0a5500642002da1f1 (DEFAULT_TABLE table=test-table [id=3c31523c0bca43fe98c2e9b97f23fda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:44.195344 19761 tablet_service.cc:1467] Processing CreateTablet for tablet 27ed9a0b72a44ee0a5500642002da1f1 (DEFAULT_TABLE table=test-table [id=3c31523c0bca43fe98c2e9b97f23fda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:44.195669 19629 tablet_service.cc:1467] Processing CreateTablet for tablet 27ed9a0b72a44ee0a5500642002da1f1 (DEFAULT_TABLE table=test-table [id=3c31523c0bca43fe98c2e9b97f23fda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:44.196435 19496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ed9a0b72a44ee0a5500642002da1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:44.196642 19761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ed9a0b72a44ee0a5500642002da1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:44.196861 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ed9a0b72a44ee0a5500642002da1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:44.208215 19844 tablet_bootstrap.cc:492] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Bootstrap starting.
I20260924 03:15:44.211480 19846 tablet_bootstrap.cc:492] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Bootstrap starting.
I20260924 03:15:44.211925 19844 tablet_bootstrap.cc:654] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:44.212236 19845 tablet_bootstrap.cc:492] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Bootstrap starting.
I20260924 03:15:44.213223 19844 log.cc:824] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:44.215426 19846 tablet_bootstrap.cc:654] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:44.216059 19845 tablet_bootstrap.cc:654] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:44.216450 19846 log.cc:824] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:44.217461 19845 log.cc:824] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:44.219376 19844 tablet_bootstrap.cc:492] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: No bootstrap required, opened a new log
I20260924 03:15:44.219420 19846 tablet_bootstrap.cc:492] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: No bootstrap required, opened a new log
I20260924 03:15:44.219652 19844 ts_tablet_manager.cc:1397] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260924 03:15:44.219689 19846 ts_tablet_manager.cc:1397] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 03:15:44.220301 19845 tablet_bootstrap.cc:492] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: No bootstrap required, opened a new log
I20260924 03:15:44.220533 19845 ts_tablet_manager.cc:1397] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 03:15:44.226888 19844 raft_consensus.cc:348] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.227259 19844 raft_consensus.cc:374] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:44.227361 19844 raft_consensus.cc:729] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50567ad4ca3b40a5ab4866ce64b20164, State: Initialized, Role: FOLLOWER
I20260924 03:15:44.228003 19844 consensus_queue.cc:260] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.229122 19826 heartbeater.cc:503] Master 127.14.227.190:46079 was elected leader, sending a full tablet report...
I20260924 03:15:44.229619 19844 ts_tablet_manager.cc:1428] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260924 03:15:44.230756 19845 raft_consensus.cc:348] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.231199 19845 raft_consensus.cc:374] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:44.231323 19845 raft_consensus.cc:729] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be50e5ed65144913ad25501d4564d3e3, State: Initialized, Role: FOLLOWER
I20260924 03:15:44.231549 19846 raft_consensus.cc:348] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.231938 19846 raft_consensus.cc:374] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:44.232026 19846 raft_consensus.cc:729] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c06458976447dea32b3e5c7f7cec59, State: Initialized, Role: FOLLOWER
I20260924 03:15:44.231920 19845 consensus_queue.cc:260] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.232625 19846 consensus_queue.cc:260] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.232962 19694 heartbeater.cc:503] Master 127.14.227.190:46079 was elected leader, sending a full tablet report...
I20260924 03:15:44.233740 19845 ts_tablet_manager.cc:1428] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260924 03:15:44.236129 19846 ts_tablet_manager.cc:1428] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260924 03:15:44.341583 19827 tablet.cc:2369] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:15:44.348433 19695 tablet.cc:2369] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:44.367654 19852 raft_consensus.cc:482] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:44.367919 19852 raft_consensus.cc:504] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.368556 19850 raft_consensus.cc:482] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:44.368822 19850 raft_consensus.cc:504] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.369853 19852 leader_election.cc:288] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50567ad4ca3b40a5ab4866ce64b20164 (127.14.227.131:39357), be50e5ed65144913ad25501d4564d3e3 (127.14.227.130:43437)
I20260924 03:15:44.372468 19850 leader_election.cc:288] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6c06458976447dea32b3e5c7f7cec59 (127.14.227.129:39669), be50e5ed65144913ad25501d4564d3e3 (127.14.227.130:43437)
W20260924 03:15:44.373541 19562 tablet.cc:2369] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:44.380544 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.370348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:33921 (local address 127.14.227.130:43437)
0924 03:15:44.370580 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.370590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.371597 (+ 1007us) server_negotiation.cc:407] Connection header received
0924 03:15:44.371917 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.371925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.372021 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.372085 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:44.372904 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.374548 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.378270 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.379405 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.379553 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.379642 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.380129 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.380257 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:15:44.380687 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.370757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39669 (local address 127.14.227.131:43553)
0924 03:15:44.371560 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:44.371603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:44.371740 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:44.374593 (+ 2853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:44.374600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:44.374633 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:44.374923 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.374936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.377159 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.377162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.378400 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.378413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.379946 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.379962 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.380179 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.380233 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 03:15:44.380310 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:15:44.380436 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":4}
I20260924 03:15:44.381170 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.370096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:39357 (local address 127.14.227.129:58815)
0924 03:15:44.372162 (+ 2066us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:44.372179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:44.372276 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:44.372836 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:44.372842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:44.372869 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:44.373193 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.373204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.374648 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.374656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.378704 (+ 4048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.378717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.380354 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.380359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.380533 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.380563 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:15:44.380636 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:15:44.380763 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":1878,"thread_start_us":1056,"threads_started":1}
I20260924 03:15:44.381278 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.370294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:43437 (local address 127.14.227.129:33921)
0924 03:15:44.371485 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:44.371507 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:44.371732 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:44.372280 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:44.372284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:44.372306 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:44.372734 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.372746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.374718 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.374722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.378117 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.378130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.379788 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.379793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.379954 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.379981 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 03:15:44.380698 (+ 717us) client_negotiation.cc:240] Negotiation successful
0924 03:15:44.380836 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":888,"thread_start_us":480,"threads_started":1}
I20260924 03:15:44.381920 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.371586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:43553 (local address 127.14.227.129:39669)
0924 03:15:44.371750 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.371756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.371774 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:15:44.371917 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.371951 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.372083 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.374395 (+ 2312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:44.375405 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.376929 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.378586 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.379658 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.379798 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.379868 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.381599 (+ 1731us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.381711 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":53}
I20260924 03:15:44.382421 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.371041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:58815 (local address 127.14.227.131:39357)
0924 03:15:44.372569 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.372576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.372596 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:15:44.372663 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.372667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.372735 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.372828 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:44.373411 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.374398 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.378860 (+ 4462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.380068 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.380181 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.382062 (+ 1881us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.382116 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.382241 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1804,"server-negotiator.queue_time_us":204}
I20260924 03:15:44.382340 19649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ed9a0b72a44ee0a5500642002da1f1" candidate_uuid: "c6c06458976447dea32b3e5c7f7cec59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be50e5ed65144913ad25501d4564d3e3" is_pre_election: true
I20260924 03:15:44.382874 19649 raft_consensus.cc:2454] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c06458976447dea32b3e5c7f7cec59 in term 0.
I20260924 03:15:44.383002 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.372011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:43437 (local address 127.14.227.131:53443)
0924 03:15:44.375233 (+ 3222us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:44.375250 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:44.375358 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:44.376527 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:44.376534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:44.376562 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:44.376887 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.376896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.378413 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.378418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.380370 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:44.380381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.382169 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.382176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:44.382348 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.382385 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:15:44.382485 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:15:44.382621 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3112,"thread_start_us":2951,"threads_started":1}
I20260924 03:15:44.383255 19781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ed9a0b72a44ee0a5500642002da1f1" candidate_uuid: "c6c06458976447dea32b3e5c7f7cec59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50567ad4ca3b40a5ab4866ce64b20164" is_pre_election: true
I20260924 03:15:44.383256 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ed9a0b72a44ee0a5500642002da1f1" candidate_uuid: "50567ad4ca3b40a5ab4866ce64b20164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c06458976447dea32b3e5c7f7cec59" is_pre_election: true
I20260924 03:15:44.383663 19516 raft_consensus.cc:2454] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50567ad4ca3b40a5ab4866ce64b20164 in term 0.
I20260924 03:15:44.383737 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.372356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:53443 (local address 127.14.227.130:43437)
0924 03:15:44.376273 (+ 3917us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.376277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.376297 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:15:44.376337 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.376341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.376391 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.376503 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:44.377099 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.378266 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.380547 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.381732 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.381971 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.382040 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.383289 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.383417 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3814,"thread_start_us":3612,"threads_started":1}
I20260924 03:15:44.383663 19781 raft_consensus.cc:2454] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c06458976447dea32b3e5c7f7cec59 in term 0.
I20260924 03:15:44.383945 19450 leader_election.cc:302] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be50e5ed65144913ad25501d4564d3e3, c6c06458976447dea32b3e5c7f7cec59; no voters:
I20260924 03:15:44.384276 19649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ed9a0b72a44ee0a5500642002da1f1" candidate_uuid: "50567ad4ca3b40a5ab4866ce64b20164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be50e5ed65144913ad25501d4564d3e3" is_pre_election: true
I20260924 03:15:44.384253 19715 leader_election.cc:302] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50567ad4ca3b40a5ab4866ce64b20164, c6c06458976447dea32b3e5c7f7cec59; no voters:
I20260924 03:15:44.384338 19852 raft_consensus.cc:2781] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:44.384449 19649 raft_consensus.cc:2454] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50567ad4ca3b40a5ab4866ce64b20164 in term 0.
I20260924 03:15:44.384483 19852 raft_consensus.cc:482] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:44.384591 19852 raft_consensus.cc:3037] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:44.384681 19850 raft_consensus.cc:2781] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:44.384799 19850 raft_consensus.cc:482] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:44.384889 19850 raft_consensus.cc:3037] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:44.387758 19852 raft_consensus.cc:504] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.387893 19850 raft_consensus.cc:504] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.388465 19850 leader_election.cc:288] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [CANDIDATE]: Term 1 election: Requested vote from peers c6c06458976447dea32b3e5c7f7cec59 (127.14.227.129:39669), be50e5ed65144913ad25501d4564d3e3 (127.14.227.130:43437)
I20260924 03:15:44.388495 19852 leader_election.cc:288] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [CANDIDATE]: Term 1 election: Requested vote from peers 50567ad4ca3b40a5ab4866ce64b20164 (127.14.227.131:39357), be50e5ed65144913ad25501d4564d3e3 (127.14.227.130:43437)
I20260924 03:15:44.388938 19649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ed9a0b72a44ee0a5500642002da1f1" candidate_uuid: "50567ad4ca3b40a5ab4866ce64b20164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be50e5ed65144913ad25501d4564d3e3"
I20260924 03:15:44.388985 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ed9a0b72a44ee0a5500642002da1f1" candidate_uuid: "50567ad4ca3b40a5ab4866ce64b20164" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c06458976447dea32b3e5c7f7cec59"
I20260924 03:15:44.389045 19781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ed9a0b72a44ee0a5500642002da1f1" candidate_uuid: "c6c06458976447dea32b3e5c7f7cec59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50567ad4ca3b40a5ab4866ce64b20164"
I20260924 03:15:44.389164 19649 raft_consensus.cc:3037] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:44.389252 19516 raft_consensus.cc:2379] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 50567ad4ca3b40a5ab4866ce64b20164 in current term 1: Already voted for candidate c6c06458976447dea32b3e5c7f7cec59 in this term.
I20260924 03:15:44.389271 19781 raft_consensus.cc:2379] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c6c06458976447dea32b3e5c7f7cec59 in current term 1: Already voted for candidate 50567ad4ca3b40a5ab4866ce64b20164 in this term.
I20260924 03:15:44.389132 19648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ed9a0b72a44ee0a5500642002da1f1" candidate_uuid: "c6c06458976447dea32b3e5c7f7cec59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be50e5ed65144913ad25501d4564d3e3"
I20260924 03:15:44.390133 19452 leader_election.cc:302] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c06458976447dea32b3e5c7f7cec59; no voters: 50567ad4ca3b40a5ab4866ce64b20164, be50e5ed65144913ad25501d4564d3e3
I20260924 03:15:44.390503 19852 raft_consensus.cc:2726] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 03:15:44.391734 19649 raft_consensus.cc:2454] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50567ad4ca3b40a5ab4866ce64b20164 in term 1.
I20260924 03:15:44.392288 19715 leader_election.cc:302] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50567ad4ca3b40a5ab4866ce64b20164, be50e5ed65144913ad25501d4564d3e3; no voters: c6c06458976447dea32b3e5c7f7cec59
I20260924 03:15:44.392550 19850 raft_consensus.cc:2781] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:44.392926 19850 raft_consensus.cc:686] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 1 LEADER]: Becoming Leader. State: Replica: 50567ad4ca3b40a5ab4866ce64b20164, State: Running, Role: LEADER
I20260924 03:15:44.393383 19850 consensus_queue.cc:237] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } }
I20260924 03:15:44.397688 19371 catalog_manager.cc:5495] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50567ad4ca3b40a5ab4866ce64b20164 (127.14.227.131). New cstate: current_term: 1 leader_uuid: "50567ad4ca3b40a5ab4866ce64b20164" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50567ad4ca3b40a5ab4866ce64b20164" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 } health_report { overall_health: UNKNOWN } } }
I20260924 03:15:44.432205 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.427249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.14.227.131:39357)
0924 03:15:44.427407 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.427411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.427428 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:15:44.427609 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.427613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.427685 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.427803 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:15:44.428102 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.428882 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.430206 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.430603 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.430695 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.431118 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:15:44.431624 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:15:44.431733 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.431814 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.431878 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:15:44.440368 19649 raft_consensus.cc:1261] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 1 FOLLOWER]: Refusing update from remote peer 50567ad4ca3b40a5ab4866ce64b20164: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:44.440500 19516 raft_consensus.cc:1261] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 1 FOLLOWER]: Refusing update from remote peer 50567ad4ca3b40a5ab4866ce64b20164: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:44.441293 19857 consensus_queue.cc:1035] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be50e5ed65144913ad25501d4564d3e3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:44.441681 19850 consensus_queue.cc:1035] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c06458976447dea32b3e5c7f7cec59" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:44.452672 19862 mvcc.cc:205] Tried to move back new op lower bound from 7332740073217355776 to 7332740073039630336. Current Snapshot: MvccSnapshot[applied={T|T < 7332740073217355776}]
I20260924 03:15:44.467939 19371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:44.468245 19371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:44.471789 19371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 130d2116f4da4c5abf4e499d77e441d0: Sending DeleteTablet for 3 replicas of tablet 27ed9a0b72a44ee0a5500642002da1f1
I20260924 03:15:44.472961 19761 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ed9a0b72a44ee0a5500642002da1f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:44 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:43280
I20260924 03:15:44.473096 19629 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ed9a0b72a44ee0a5500642002da1f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:44 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:37246
I20260924 03:15:44.473305 19496 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ed9a0b72a44ee0a5500642002da1f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:44 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:48008
I20260924 03:15:44.475032 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 19434
I20260924 03:15:44.489678 19872 tablet_replica.cc:330] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: stopping tablet replica
I20260924 03:15:44.490216 19872 raft_consensus.cc:2229] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:44.490741 19872 raft_consensus.cc:2258] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:44.491456 19870 tablet_replica.cc:330] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: stopping tablet replica
I20260924 03:15:44.492113 19870 raft_consensus.cc:2229] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:15:44.493083 19872 ts_tablet_manager.cc:1905] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:44.497217 19870 raft_consensus.cc:2258] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:44.497912 19871 tablet_replica.cc:330] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: stopping tablet replica
I20260924 03:15:44.498404 19871 raft_consensus.cc:2229] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:44.499882 19871 raft_consensus.cc:2258] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:44.500366 19872 ts_tablet_manager.cc:1918] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:15:44.500531 19872 log.cc:1196] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/27ed9a0b72a44ee0a5500642002da1f1
I20260924 03:15:44.501207 19870 ts_tablet_manager.cc:1905] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:44.501348 19872 ts_tablet_manager.cc:1939] T 27ed9a0b72a44ee0a5500642002da1f1 P c6c06458976447dea32b3e5c7f7cec59: Deleting consensus metadata
I20260924 03:15:44.503222 19358 catalog_manager.cc:4841] TS c6c06458976447dea32b3e5c7f7cec59 (127.14.227.129:39669): tablet 27ed9a0b72a44ee0a5500642002da1f1 (table test-table [id=3c31523c0bca43fe98c2e9b97f23fda7]) successfully deleted
Sep 24 03:15:44 dist-test-slave-vd0p krb5kdc[19329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:44.504685 19871 ts_tablet_manager.cc:1905] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:44.510252 19870 ts_tablet_manager.cc:1918] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:15:44.510432 19870 log.cc:1196] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/27ed9a0b72a44ee0a5500642002da1f1
I20260924 03:15:44.511013 19870 ts_tablet_manager.cc:1939] T 27ed9a0b72a44ee0a5500642002da1f1 P 50567ad4ca3b40a5ab4866ce64b20164: Deleting consensus metadata
I20260924 03:15:44.511644 19871 ts_tablet_manager.cc:1918] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:15:44.511793 19871 log.cc:1196] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/27ed9a0b72a44ee0a5500642002da1f1
I20260924 03:15:44.512287 19871 ts_tablet_manager.cc:1939] T 27ed9a0b72a44ee0a5500642002da1f1 P be50e5ed65144913ad25501d4564d3e3: Deleting consensus metadata
I20260924 03:15:44.512862 19360 catalog_manager.cc:4841] TS 50567ad4ca3b40a5ab4866ce64b20164 (127.14.227.131:39357): tablet 27ed9a0b72a44ee0a5500642002da1f1 (table test-table [id=3c31523c0bca43fe98c2e9b97f23fda7]) successfully deleted
I20260924 03:15:44.513235 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.476292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.14.227.129:39669)
0924 03:15:44.476455 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.476461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.482513 (+ 6052us) server_negotiation.cc:407] Connection header received
0924 03:15:44.482716 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.482724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.482844 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.482968 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:44.492446 (+ 9478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.493948 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.497820 (+ 3872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.498523 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.498639 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.504825 (+ 6186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:44.504841 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:44.504860 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:44.504917 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:44.508849 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.510174 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.510184 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.510187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.510427 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.510961 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.510966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.510969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.511502 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:44.511837 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.512557 (+ 720us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.512798 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:15:44.514187 19358 catalog_manager.cc:4841] TS be50e5ed65144913ad25501d4564d3e3 (127.14.227.130:43437): tablet 27ed9a0b72a44ee0a5500642002da1f1 (table test-table [id=3c31523c0bca43fe98c2e9b97f23fda7]) successfully deleted
I20260924 03:15:44.519328 19556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:44.626396 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 19434
I20260924 03:15:44.648242 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 19567
Sep 24 03:15:44 dist-test-slave-vd0p krb5kdc[19329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:15:44.663376 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.649050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.14.227.130:43437)
0924 03:15:44.649239 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.649245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.649381 (+ 136us) server_negotiation.cc:407] Connection header received
0924 03:15:44.649594 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.649598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.649660 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.649764 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:44.651003 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.651718 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.652951 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.653347 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.653491 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.657608 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:44.657619 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:44.657632 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:44.657660 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:44.660659 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.661268 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.661283 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.661286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.661401 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.661751 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.661756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.661760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.662225 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:44.662533 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.662793 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.663031 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:15:44.663944 19689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:44.753733 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 19567
I20260924 03:15:44.770670 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 19699
Sep 24 03:15:44 dist-test-slave-vd0p krb5kdc[19329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:15:44.788028 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:44.771430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.14.227.131:39357)
0924 03:15:44.771639 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:44.771645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:44.771668 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:15:44.771808 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:44.771814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:44.771914 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:44.772029 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:44.773317 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.774521 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.775745 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:44.776431 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:44.776574 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:44.780985 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:44.781000 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:44.781020 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:44.781062 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:44.785187 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.785874 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.785883 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.785887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.786077 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:44.786431 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:44.786436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:44.786440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:44.786925 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:44.787243 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:44.787419 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 03:15:44.787651 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:15:44.788810 19821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:44.908825 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 19699
I20260924 03:15:44.927350 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 19342
I20260924 03:15:44.928455 19403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:45.033778 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 19342
2026-09-24T03:15:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4200 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T03:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:45Z Disabled control of system clock
I20260924 03:15:45.096480 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33525
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:33525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:15:45.409219 19881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:45.409490 19881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:45.409570 19881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:45.416507 19881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:45.416600 19881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:45.416635 19881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:45.416682 19881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:45.416715 19881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:45.425642 19881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:33525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:45.426852 19881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:45.428491 19881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:45.440047 19889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:45.440128 19886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:45.440107 19887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:45.440892 19881 server_base.cc:1034] running on GCE node
I20260924 03:15:45.441617 19881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:45.442991 19881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:45.444223 19881 hybrid_clock.cc:648] HybridClock initialized: now 1790219745444175 us; error 52 us; skew 500 ppm
I20260924 03:15:45.444563 19881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:15:45.446713 19881 webserver.cc:460] Webserver started at http://127.14.227.190:35329/ using document root <none> and password file <none>
I20260924 03:15:45.447381 19881 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:45.447510 19881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:45.447818 19881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:45.450546 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "b9e6766ca2064c35ae63b8f5e06bbaae"
format_stamp: "Formatted at 2026-09-24 03:15:45 on dist-test-slave-vd0p"
server_key: "3d1c8f6d1555cf7cd65be4d09b7bced5"
server_key_iv: "6f90cac7bc75621f8ba93bf0653f43c6"
server_key_version: "encryptionkey@0"
I20260924 03:15:45.451550 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "b9e6766ca2064c35ae63b8f5e06bbaae"
format_stamp: "Formatted at 2026-09-24 03:15:45 on dist-test-slave-vd0p"
server_key: "3d1c8f6d1555cf7cd65be4d09b7bced5"
server_key_iv: "6f90cac7bc75621f8ba93bf0653f43c6"
server_key_version: "encryptionkey@0"
I20260924 03:15:45.456845 19881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:15:45.460328 19895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:45.461670 19881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 03:15:45.461880 19881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "b9e6766ca2064c35ae63b8f5e06bbaae"
format_stamp: "Formatted at 2026-09-24 03:15:45 on dist-test-slave-vd0p"
server_key: "3d1c8f6d1555cf7cd65be4d09b7bced5"
server_key_iv: "6f90cac7bc75621f8ba93bf0653f43c6"
server_key_version: "encryptionkey@0"
I20260924 03:15:45.462178 19881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:45.467713 19881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:45.468430 19881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:45.468717 19881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:45.486202 19881 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:33525
I20260924 03:15:45.486231 19946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:33525 every 8 connection(s)
I20260924 03:15:45.487476 19881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:45.491072 19947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:45.491591 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 19881
I20260924 03:15:45.491783 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:45.492424 15246 external_mini_cluster.cc:1428] Setting key 1736a5473f7fe556fc71cefab151e4ff
I20260924 03:15:45.501255 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:45.494409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.14.227.190:33525)
0924 03:15:45.495494 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:45.495508 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:45.495562 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:15:45.497466 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:45.497490 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:45.498085 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:45.498510 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:45.499429 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:45.499436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:45.499452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:45.499467 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:45.499535 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:45.499554 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:45.499755 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:45.499884 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:15:45.500122 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":360,"threads_started":1}
I20260924 03:15:45.504971 19947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae: Bootstrap starting.
I20260924 03:15:45.507928 19947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:45.508765 19947 log.cc:824] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:45.511132 19947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae: No bootstrap required, opened a new log
I20260924 03:15:45.518115 19947 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e6766ca2064c35ae63b8f5e06bbaae" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33525 } }
I20260924 03:15:45.518416 19947 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:45.518532 19947 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e6766ca2064c35ae63b8f5e06bbaae, State: Initialized, Role: FOLLOWER
I20260924 03:15:45.519052 19947 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e6766ca2064c35ae63b8f5e06bbaae" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33525 } }
I20260924 03:15:45.519239 19947 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:45.519339 19947 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:45.519470 19947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:45.521680 19947 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e6766ca2064c35ae63b8f5e06bbaae" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33525 } }
I20260924 03:15:45.522150 19947 leader_election.cc:302] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9e6766ca2064c35ae63b8f5e06bbaae; no voters:
I20260924 03:15:45.522652 19947 leader_election.cc:288] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:45.522786 19952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:45.523499 19952 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [term 1 LEADER]: Becoming Leader. State: Replica: b9e6766ca2064c35ae63b8f5e06bbaae, State: Running, Role: LEADER
I20260924 03:15:45.524051 19952 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e6766ca2064c35ae63b8f5e06bbaae" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33525 } }
I20260924 03:15:45.524677 19947 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:45.526899 19953 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9e6766ca2064c35ae63b8f5e06bbaae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e6766ca2064c35ae63b8f5e06bbaae" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33525 } } }
I20260924 03:15:45.526873 19954 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9e6766ca2064c35ae63b8f5e06bbaae. Latest consensus state: current_term: 1 leader_uuid: "b9e6766ca2064c35ae63b8f5e06bbaae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9e6766ca2064c35ae63b8f5e06bbaae" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33525 } } }
I20260924 03:15:45.527300 19953 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:45.527367 19954 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:45.529443 19959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:45.535390 19959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:45.544154 19959 catalog_manager.cc:1347] Generated new cluster ID: c8f8c0fb5c9548bbaebad60ce03a8618
I20260924 03:15:45.544258 19959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:45.564640 19959 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:45.565099 19959 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:45.575341 19959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae: Generated new TSK 0
I20260924 03:15:45.576056 19959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:15:45.587584 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33525
--builtin_ntp_servers=127.14.227.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 03:15:45.910893 19971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:45.911155 19971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:45.911334 19971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:45.918573 19971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:45.918701 19971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:45.918900 19971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:45.928450 19971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:45.929757 19971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:45.931738 19971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:45.944670 19979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:45.944561 19976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:45.944586 19977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:45.945322 19971 server_base.cc:1034] running on GCE node
I20260924 03:15:45.945986 19971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:45.946964 19971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:45.948182 19971 hybrid_clock.cc:648] HybridClock initialized: now 1790219745948134 us; error 66 us; skew 500 ppm
I20260924 03:15:45.948480 19971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:15:45.950657 19971 webserver.cc:460] Webserver started at http://127.14.227.129:37271/ using document root <none> and password file <none>
I20260924 03:15:45.951362 19971 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:45.951493 19971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:45.951810 19971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:45.954674 19971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21"
format_stamp: "Formatted at 2026-09-24 03:15:45 on dist-test-slave-vd0p"
server_key: "33435f0196add0d8e38d75b849adb308"
server_key_iv: "0635ecf5d382cfae18a1d3640744bf25"
server_key_version: "encryptionkey@0"
I20260924 03:15:45.955554 19971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21"
format_stamp: "Formatted at 2026-09-24 03:15:45 on dist-test-slave-vd0p"
server_key: "33435f0196add0d8e38d75b849adb308"
server_key_iv: "0635ecf5d382cfae18a1d3640744bf25"
server_key_version: "encryptionkey@0"
I20260924 03:15:45.960596 19971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:15:45.963914 19985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:45.965306 19971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 03:15:45.965493 19971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21"
format_stamp: "Formatted at 2026-09-24 03:15:45 on dist-test-slave-vd0p"
server_key: "33435f0196add0d8e38d75b849adb308"
server_key_iv: "0635ecf5d382cfae18a1d3640744bf25"
server_key_version: "encryptionkey@0"
I20260924 03:15:45.965705 19971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:45.971109 19971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:45.971889 19971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:45.972138 19971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:45.972987 19971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:45.974326 19971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:45.974442 19971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:45.974571 19971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:45.974658 19971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:46.007592 19971 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:46735
I20260924 03:15:46.007643 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:46735 every 8 connection(s)
I20260924 03:15:46.009040 19971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:46.017403 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.011720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:52835 (local address 127.14.227.190:33525)
0924 03:15:46.012152 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:46.012156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:46.013396 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 03:15:46.015133 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:46.015139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:46.015221 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:46.015341 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:46.016639 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:46.016644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:46.016649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:46.016667 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:46.016735 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:46.016747 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:46.016868 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:46.017051 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 03:15:46.017113 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260924 03:15:46.017812 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 19971
I20260924 03:15:46.018024 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:46.018381 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.012311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33525 (local address 127.14.227.129:52835)
0924 03:15:46.013224 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:46.013265 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:46.014873 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:46.015560 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:46.015568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:46.015656 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:46.016363 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:15:46.016366 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:46.016401 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:15:46.016479 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:46.016842 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:46.016852 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:46.016859 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:15:46.017134 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 03:15:46.017371 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":262,"threads_started":1}
I20260924 03:15:46.018785 15246 external_mini_cluster.cc:1428] Setting key 1969752bbc87faf2c9a75f9263879922
I20260924 03:15:46.020102 20098 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33525
I20260924 03:15:46.020375 20098 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:46.021001 20098 heartbeater.cc:511] Master 127.14.227.190:33525 requested a full tablet report, sending...
I20260924 03:15:46.021626 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33525
--builtin_ntp_servers=127.14.227.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:15:46.022985 19912 ts_manager.cc:194] Registered new tserver with Master: 21f1f2cb618d4b24aaaa3d8bb56e3a21 (127.14.227.129:46735)
W20260924 03:15:46.331387 20102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:46.331645 20102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:46.331869 20102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:46.338963 20102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:46.339046 20102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:46.339202 20102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:46.347878 20102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:46.349085 20102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:46.350807 20102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:46.363493 20107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:46.363554 20108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:46.363580 20110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:46.364251 20102 server_base.cc:1034] running on GCE node
I20260924 03:15:46.364781 20102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:46.365643 20102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:46.366853 20102 hybrid_clock.cc:648] HybridClock initialized: now 1790219746366814 us; error 52 us; skew 500 ppm
I20260924 03:15:46.367144 20102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:15:46.369186 20102 webserver.cc:460] Webserver started at http://127.14.227.130:42979/ using document root <none> and password file <none>
I20260924 03:15:46.369807 20102 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:46.369930 20102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:46.370265 20102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:46.372819 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "15c7f420fc6a48b29a2d9529d14a1a7f"
format_stamp: "Formatted at 2026-09-24 03:15:46 on dist-test-slave-vd0p"
server_key: "87bf596938c4cc4ad7509e1ce43bb4fa"
server_key_iv: "9e3a61865cd61858cfe38b477598ff50"
server_key_version: "encryptionkey@0"
I20260924 03:15:46.373557 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "15c7f420fc6a48b29a2d9529d14a1a7f"
format_stamp: "Formatted at 2026-09-24 03:15:46 on dist-test-slave-vd0p"
server_key: "87bf596938c4cc4ad7509e1ce43bb4fa"
server_key_iv: "9e3a61865cd61858cfe38b477598ff50"
server_key_version: "encryptionkey@0"
I20260924 03:15:46.378274 20102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260924 03:15:46.381268 20116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:46.382452 20102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 03:15:46.382616 20102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "15c7f420fc6a48b29a2d9529d14a1a7f"
format_stamp: "Formatted at 2026-09-24 03:15:46 on dist-test-slave-vd0p"
server_key: "87bf596938c4cc4ad7509e1ce43bb4fa"
server_key_iv: "9e3a61865cd61858cfe38b477598ff50"
server_key_version: "encryptionkey@0"
I20260924 03:15:46.382841 20102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:46.387990 20102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:46.388681 20102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:46.388993 20102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:46.389873 20102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:46.391139 20102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:46.391258 20102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:46.391390 20102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:46.391479 20102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:46.423117 20102 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:37039
I20260924 03:15:46.423197 20228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:37039 every 8 connection(s)
I20260924 03:15:46.424404 20102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:46.428725 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 20102
I20260924 03:15:46.428887 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:46.429468 15246 external_mini_cluster.cc:1428] Setting key ad95734312eee660fd7ab436ce119ed0
I20260924 03:15:46.432672 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33525
--builtin_ntp_servers=127.14.227.148:37003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 03:15:46.433197 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.427000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:47717 (local address 127.14.227.190:33525)
0924 03:15:46.427186 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:46.427190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:46.428533 (+ 1343us) server_negotiation.cc:407] Connection header received
0924 03:15:46.430789 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:46.430794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:46.430871 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:46.431007 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:46.432406 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:46.432414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:46.432421 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:46.432443 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:46.432486 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:46.432503 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:46.432621 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:46.432792 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 03:15:46.432857 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:15:46.434104 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.427488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33525 (local address 127.14.227.130:47717)
0924 03:15:46.428364 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:46.428410 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:46.430513 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:46.431212 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:46.431239 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:46.431357 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:46.432115 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:15:46.432118 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:46.432156 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:15:46.432240 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:46.432600 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:46.432606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:46.432611 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:15:46.432873 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 03:15:46.433116 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":263,"threads_started":1}
I20260924 03:15:46.435484 20229 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33525
I20260924 03:15:46.435773 20229 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:46.436313 20229 heartbeater.cc:511] Master 127.14.227.190:33525 requested a full tablet report, sending...
I20260924 03:15:46.437626 19912 ts_manager.cc:194] Registered new tserver with Master: 15c7f420fc6a48b29a2d9529d14a1a7f (127.14.227.130:37039)
W20260924 03:15:46.749516 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:46.749823 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:46.750002 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:46.756882 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:46.757004 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:46.757170 20232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:46.765666 20232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:46.766971 20232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:46.768741 20232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:46.781438 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:46.781659 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:46.781461 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:46.782328 20232 server_base.cc:1034] running on GCE node
I20260924 03:15:46.782866 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:46.783687 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:46.784884 20232 hybrid_clock.cc:648] HybridClock initialized: now 1790219746784849 us; error 32 us; skew 500 ppm
I20260924 03:15:46.785158 20232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:15:46.787334 20232 webserver.cc:460] Webserver started at http://127.14.227.131:45937/ using document root <none> and password file <none>
I20260924 03:15:46.787940 20232 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:46.788018 20232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:46.788293 20232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:46.791038 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "506eda2d32c0411dbe0ec244f7bec1bc"
format_stamp: "Formatted at 2026-09-24 03:15:46 on dist-test-slave-vd0p"
server_key: "7ce7a840195f512287c05ce4a0ac1095"
server_key_iv: "bab89fa321bee5c4c8912f76eb6e51e9"
server_key_version: "encryptionkey@0"
I20260924 03:15:46.791769 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "506eda2d32c0411dbe0ec244f7bec1bc"
format_stamp: "Formatted at 2026-09-24 03:15:46 on dist-test-slave-vd0p"
server_key: "7ce7a840195f512287c05ce4a0ac1095"
server_key_iv: "bab89fa321bee5c4c8912f76eb6e51e9"
server_key_version: "encryptionkey@0"
I20260924 03:15:46.796446 20232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:15:46.799721 20246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:46.800987 20232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:15:46.801112 20232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "506eda2d32c0411dbe0ec244f7bec1bc"
format_stamp: "Formatted at 2026-09-24 03:15:46 on dist-test-slave-vd0p"
server_key: "7ce7a840195f512287c05ce4a0ac1095"
server_key_iv: "bab89fa321bee5c4c8912f76eb6e51e9"
server_key_version: "encryptionkey@0"
I20260924 03:15:46.801272 20232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:46.806408 20232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:46.807063 20232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:46.807271 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:46.808012 20232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:46.809244 20232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:46.809325 20232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:46.809399 20232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:46.809451 20232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:46.841564 20232 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:39207
I20260924 03:15:46.841637 20358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:39207 every 8 connection(s)
I20260924 03:15:46.842893 20232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:46.850490 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.845329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:40251 (local address 127.14.227.190:33525)
0924 03:15:46.845530 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:46.845537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:46.846935 (+ 1398us) server_negotiation.cc:407] Connection header received
0924 03:15:46.848507 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:46.848511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:46.848560 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:46.848656 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:46.849896 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:46.849899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:46.849902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:46.849914 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:46.849937 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:46.849990 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:46.850087 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:46.850240 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 03:15:46.850286 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:15:46.851015 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 20232
I20260924 03:15:46.851164 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:46.851418 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.845857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33525 (local address 127.14.227.131:40251)
0924 03:15:46.846792 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:46.846826 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:46.848271 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:46.848838 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:46.848846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:46.848930 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:46.849629 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:15:46.849632 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:46.849667 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:15:46.849745 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:46.850067 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:46.850073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:46.850078 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:15:46.850271 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 03:15:46.850452 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":248,"threads_started":1}
I20260924 03:15:46.851651 15246 external_mini_cluster.cc:1428] Setting key 56cd826a33757b08adea76ce8a863abf
I20260924 03:15:46.852871 20359 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33525
I20260924 03:15:46.853166 20359 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:46.853821 20359 heartbeater.cc:511] Master 127.14.227.190:33525 requested a full tablet report, sending...
I20260924 03:15:46.855130 19912 ts_manager.cc:194] Registered new tserver with Master: 506eda2d32c0411dbe0ec244f7bec1bc (127.14.227.131:39207)
I20260924 03:15:46.866624 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:46.873461 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.871018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.14.227.190:33525)
0924 03:15:46.871185 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:46.871189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:46.871670 (+ 481us) server_negotiation.cc:407] Connection header received
0924 03:15:46.871862 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:46.871867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:46.871928 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:46.872021 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:46.872835 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:46.872840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:46.872845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:46.872861 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:46.872908 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:46.872919 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:46.873046 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:46.873142 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 03:15:46.873207 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:15:46.876956 19912 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:15:46.883859 19912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:15:46.902114 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.895389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:39207 (local address 127.0.0.1:44840)
0924 03:15:46.897507 (+ 2118us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:46.897536 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:46.897697 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:46.899548 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:46.899555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:46.899646 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:46.900640 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:15:46.900644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:46.900680 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:15:46.900765 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:46.901318 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:46.901328 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:46.901337 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:15:46.901521 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 03:15:46.901604 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1927,"thread_start_us":2422,"threads_started":1}
I20260924 03:15:46.902169 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.896055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.14.227.129:46735)
0924 03:15:46.896641 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:46.896654 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:46.898290 (+ 1636us) server_negotiation.cc:407] Connection header received
0924 03:15:46.899372 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:46.899381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:46.899584 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:46.899713 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:46.901211 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:46.901218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:46.901231 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:46.901253 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:46.901308 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:46.901328 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:46.901426 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:46.901706 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 03:15:46.901811 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":288,"threads_started":1}
I20260924 03:15:46.903028 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.895512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.14.227.131:39207)
0924 03:15:46.897165 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:46.897174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:46.897660 (+ 486us) server_negotiation.cc:407] Connection header received
0924 03:15:46.897938 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:46.897947 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:46.898254 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:46.898401 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:46.901076 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:46.901088 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:46.901104 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:46.901125 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:46.901184 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:46.901206 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:46.901372 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:46.901546 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 03:15:46.902696 (+ 1150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":1330,"threads_started":1}
I20260924 03:15:46.903266 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.895560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51618)
0924 03:15:46.898123 (+ 2563us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:46.898138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:46.899203 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:46.900156 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:46.900163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:46.900202 (+ 39us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:46.900976 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:15:46.900980 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:46.900996 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:15:46.901038 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:46.901520 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:46.901524 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:46.901528 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:15:46.902782 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0924 03:15:46.902834 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2427,"thread_start_us":2071,"threads_started":1}
I20260924 03:15:46.906946 20294 tablet_service.cc:1467] Processing CreateTablet for tablet 5e906c95388b444ba44ad99faf7ae821 (DEFAULT_TABLE table=test-table [id=a447159c520e4546a21ef355541d72c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:46.906945 20033 tablet_service.cc:1467] Processing CreateTablet for tablet 5e906c95388b444ba44ad99faf7ae821 (DEFAULT_TABLE table=test-table [id=a447159c520e4546a21ef355541d72c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:46.908370 20294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e906c95388b444ba44ad99faf7ae821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:46.908370 20033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e906c95388b444ba44ad99faf7ae821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:46.914356 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.898784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.14.227.130:37039)
0924 03:15:46.900964 (+ 2180us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:46.900973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:46.901438 (+ 465us) server_negotiation.cc:407] Connection header received
0924 03:15:46.901609 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:46.901616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:46.901853 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:46.902020 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:46.913481 (+ 11461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:46.913489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:46.913499 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:46.913515 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:46.913560 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:46.913575 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:46.913783 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:46.913864 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 03:15:46.913983 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996,"thread_start_us":1908,"threads_started":1}
I20260924 03:15:46.914392 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:46.898436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:37039 (local address 127.0.0.1:59816)
0924 03:15:46.901326 (+ 2890us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:46.901342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:46.901435 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:46.902339 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:46.902347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:46.902372 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:46.913259 (+ 10887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:15:46.913262 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:46.913275 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:15:46.913313 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:46.913707 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:46.913712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:46.913717 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:15:46.913841 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 03:15:46.913897 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2752,"thread_start_us":2639,"threads_started":1}
I20260924 03:15:46.917927 20164 tablet_service.cc:1467] Processing CreateTablet for tablet 5e906c95388b444ba44ad99faf7ae821 (DEFAULT_TABLE table=test-table [id=a447159c520e4546a21ef355541d72c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:15:46.919148 20164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e906c95388b444ba44ad99faf7ae821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:46.923074 20377 tablet_bootstrap.cc:492] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Bootstrap starting.
I20260924 03:15:46.925298 20378 tablet_bootstrap.cc:492] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21: Bootstrap starting.
I20260924 03:15:46.927604 20377 tablet_bootstrap.cc:654] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:46.928577 20377 log.cc:824] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:46.928815 20378 tablet_bootstrap.cc:654] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:46.930011 20378 log.cc:824] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:46.931697 20377 tablet_bootstrap.cc:492] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: No bootstrap required, opened a new log
I20260924 03:15:46.931967 20377 ts_tablet_manager.cc:1397] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 03:15:46.933879 20378 tablet_bootstrap.cc:492] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21: No bootstrap required, opened a new log
I20260924 03:15:46.934257 20378 ts_tablet_manager.cc:1397] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260924 03:15:46.938494 20381 tablet_bootstrap.cc:492] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: Bootstrap starting.
I20260924 03:15:46.939325 20377 raft_consensus.cc:348] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:46.939766 20377 raft_consensus.cc:374] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:46.939863 20377 raft_consensus.cc:729] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506eda2d32c0411dbe0ec244f7bec1bc, State: Initialized, Role: FOLLOWER
I20260924 03:15:46.940475 20377 consensus_queue.cc:260] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:46.941512 20359 heartbeater.cc:503] Master 127.14.227.190:33525 was elected leader, sending a full tablet report...
I20260924 03:15:46.942256 20381 tablet_bootstrap.cc:654] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:46.943146 20377 ts_tablet_manager.cc:1428] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260924 03:15:46.943435 20381 log.cc:824] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:46.946004 20381 tablet_bootstrap.cc:492] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: No bootstrap required, opened a new log
I20260924 03:15:46.946256 20381 ts_tablet_manager.cc:1397] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260924 03:15:46.947146 20378 raft_consensus.cc:348] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:46.947588 20378 raft_consensus.cc:374] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:46.947722 20378 raft_consensus.cc:729] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f1f2cb618d4b24aaaa3d8bb56e3a21, State: Initialized, Role: FOLLOWER
I20260924 03:15:46.948302 20378 consensus_queue.cc:260] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:46.949318 20098 heartbeater.cc:503] Master 127.14.227.190:33525 was elected leader, sending a full tablet report...
I20260924 03:15:46.949790 20378 ts_tablet_manager.cc:1428] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
I20260924 03:15:46.954723 20381 raft_consensus.cc:348] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:46.955024 20381 raft_consensus.cc:374] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:46.955090 20381 raft_consensus.cc:729] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c7f420fc6a48b29a2d9529d14a1a7f, State: Initialized, Role: FOLLOWER
I20260924 03:15:46.955545 20381 consensus_queue.cc:260] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:46.956595 20229 heartbeater.cc:503] Master 127.14.227.190:33525 was elected leader, sending a full tablet report...
I20260924 03:15:46.957113 20381 ts_tablet_manager.cc:1428] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
W20260924 03:15:47.011469 20099 tablet.cc:2369] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:47.052209 20382 raft_consensus.cc:482] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:15:47.052472 20382 raft_consensus.cc:504] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:47.054698 20382 leader_election.cc:288] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21f1f2cb618d4b24aaaa3d8bb56e3a21 (127.14.227.129:46735), 15c7f420fc6a48b29a2d9529d14a1a7f (127.14.227.130:37039)
I20260924 03:15:47.058162 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.055244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:58947 (local address 127.14.227.129:46735)
0924 03:15:47.055418 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:47.055422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:47.056146 (+ 724us) server_negotiation.cc:407] Connection header received
0924 03:15:47.056243 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:47.056249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:47.056320 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:47.056423 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:47.057504 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:47.057511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:47.057517 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:47.057539 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:47.057580 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:47.057600 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:47.057721 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:47.057822 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 03:15:47.057880 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:15:47.058152 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.055038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.14.227.131:58947)
0924 03:15:47.055973 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.055994 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:47.056120 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:47.056449 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:47.056457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:47.056482 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:47.057271 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:15:47.057277 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:47.057303 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:15:47.057360 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:47.057714 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:47.057719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:47.057723 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:15:47.057825 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:15:47.057876 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793}
I20260924 03:15:47.058869 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.055800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:59267 (local address 127.14.227.130:37039)
0924 03:15:47.056017 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:47.056025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:47.056372 (+ 347us) server_negotiation.cc:407] Connection header received
0924 03:15:47.056593 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:47.056606 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:47.056685 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:47.056756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:47.057922 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:47.057929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:47.057933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:47.057989 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:47.058054 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:47.058117 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:47.058309 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:47.058544 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 03:15:47.058620 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:15:47.059363 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e906c95388b444ba44ad99faf7ae821" candidate_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" is_pre_election: true
I20260924 03:15:47.059665 20386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.055650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:37039 (local address 127.14.227.131:59267)
0924 03:15:47.056273 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.056289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:47.056446 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:47.056976 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:47.056981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:47.057005 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:47.057679 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:15:47.057685 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:47.057703 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:15:47.057745 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:47.058398 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:47.058405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:47.058412 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 03:15:47.059057 (+ 645us) client_negotiation.cc:240] Negotiation successful
0924 03:15:47.059109 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":541,"threads_started":1}
I20260924 03:15:47.059971 20053 raft_consensus.cc:2454] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506eda2d32c0411dbe0ec244f7bec1bc in term 0.
I20260924 03:15:47.060632 20250 leader_election.cc:302] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f1f2cb618d4b24aaaa3d8bb56e3a21, 506eda2d32c0411dbe0ec244f7bec1bc; no voters:
I20260924 03:15:47.061201 20382 raft_consensus.cc:2781] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:15:47.061353 20382 raft_consensus.cc:482] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:15:47.061458 20382 raft_consensus.cc:3037] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:47.061406 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e906c95388b444ba44ad99faf7ae821" candidate_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" is_pre_election: true
I20260924 03:15:47.062002 20184 raft_consensus.cc:2454] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506eda2d32c0411dbe0ec244f7bec1bc in term 0.
I20260924 03:15:47.064546 20382 raft_consensus.cc:504] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:47.065351 20382 leader_election.cc:288] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [CANDIDATE]: Term 1 election: Requested vote from peers 21f1f2cb618d4b24aaaa3d8bb56e3a21 (127.14.227.129:46735), 15c7f420fc6a48b29a2d9529d14a1a7f (127.14.227.130:37039)
I20260924 03:15:47.065775 20053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e906c95388b444ba44ad99faf7ae821" candidate_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21"
I20260924 03:15:47.065913 20184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e906c95388b444ba44ad99faf7ae821" candidate_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f"
I20260924 03:15:47.066013 20053 raft_consensus.cc:3037] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:47.066133 20184 raft_consensus.cc:3037] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:47.069263 20184 raft_consensus.cc:2454] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506eda2d32c0411dbe0ec244f7bec1bc in term 1.
I20260924 03:15:47.069269 20053 raft_consensus.cc:2454] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506eda2d32c0411dbe0ec244f7bec1bc in term 1.
I20260924 03:15:47.069686 20250 leader_election.cc:302] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c7f420fc6a48b29a2d9529d14a1a7f, 506eda2d32c0411dbe0ec244f7bec1bc; no voters:
I20260924 03:15:47.069989 20382 raft_consensus.cc:2781] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:47.070467 20382 raft_consensus.cc:686] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 1 LEADER]: Becoming Leader. State: Replica: 506eda2d32c0411dbe0ec244f7bec1bc, State: Running, Role: LEADER
I20260924 03:15:47.071130 20382 consensus_queue.cc:237] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } }
I20260924 03:15:47.079188 19912 catalog_manager.cc:5495] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 506eda2d32c0411dbe0ec244f7bec1bc (127.14.227.131). New cstate: current_term: 1 leader_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506eda2d32c0411dbe0ec244f7bec1bc" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 } health_report { overall_health: UNKNOWN } } }
I20260924 03:15:47.093102 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.090849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.14.227.131:39207)
0924 03:15:47.091190 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:47.091196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:47.091215 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:15:47.091345 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:47.091350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:47.091439 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:47.091545 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:47.092473 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:47.092478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:47.092483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:47.092500 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:47.092538 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:47.092550 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:47.092674 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:47.092762 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 03:15:47.092821 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
W20260924 03:15:47.094553 20360 tablet.cc:2369] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:15:47.102612 20184 raft_consensus.cc:1261] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 1 FOLLOWER]: Refusing update from remote peer 506eda2d32c0411dbe0ec244f7bec1bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:47.102799 20053 raft_consensus.cc:1261] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 1 FOLLOWER]: Refusing update from remote peer 506eda2d32c0411dbe0ec244f7bec1bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:15:47.103705 20387 consensus_queue.cc:1035] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c7f420fc6a48b29a2d9529d14a1a7f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:47.104130 20382 consensus_queue.cc:1035] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f1f2cb618d4b24aaaa3d8bb56e3a21" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:15:47.128072 19911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:47.128382 19911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:15:47.132191 19911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9e6766ca2064c35ae63b8f5e06bbaae: Sending DeleteTablet for 3 replicas of tablet 5e906c95388b444ba44ad99faf7ae821
I20260924 03:15:47.133458 20294 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e906c95388b444ba44ad99faf7ae821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:47 UTC) from {username='slave'} at 127.0.0.1:44840
I20260924 03:15:47.133590 20033 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e906c95388b444ba44ad99faf7ae821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:47 UTC) from {username='slave'} at 127.0.0.1:51618
I20260924 03:15:47.133967 20164 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e906c95388b444ba44ad99faf7ae821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:15:47 UTC) from {username='slave'} at 127.0.0.1:59816
I20260924 03:15:47.137017 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 19971
I20260924 03:15:47.147274 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.138969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.14.227.129:46735)
0924 03:15:47.139218 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:47.139229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:47.142471 (+ 3242us) server_negotiation.cc:407] Connection header received
0924 03:15:47.143460 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:47.143470 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:47.143616 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:47.143720 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:47.145900 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:47.145911 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:47.145920 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:47.145974 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:47.146031 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:47.146055 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:47.146188 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:47.146698 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 03:15:47.146783 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:15:47.148916 20093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:47.151362 20401 tablet_replica.cc:330] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: stopping tablet replica
I20260924 03:15:47.151979 20401 raft_consensus.cc:2229] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:47.152683 20401 raft_consensus.cc:2258] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:47.155913 20401 ts_tablet_manager.cc:1905] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:47.165540 20401 ts_tablet_manager.cc:1918] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:15:47.165735 20401 log.cc:1196] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/5e906c95388b444ba44ad99faf7ae821
I20260924 03:15:47.166544 20401 ts_tablet_manager.cc:1939] T 5e906c95388b444ba44ad99faf7ae821 P 15c7f420fc6a48b29a2d9529d14a1a7f: Deleting consensus metadata
I20260924 03:15:47.168692 19899 catalog_manager.cc:4841] TS 15c7f420fc6a48b29a2d9529d14a1a7f (127.14.227.130:37039): tablet 5e906c95388b444ba44ad99faf7ae821 (table test-table [id=a447159c520e4546a21ef355541d72c7]) successfully deleted
I20260924 03:15:47.168944 20399 tablet_replica.cc:330] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: stopping tablet replica
I20260924 03:15:47.169463 20399 raft_consensus.cc:2229] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:15:47.170141 20399 raft_consensus.cc:2258] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:47.171892 20399 ts_tablet_manager.cc:1905] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:47.177362 20399 ts_tablet_manager.cc:1918] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:15:47.177547 20399 log.cc:1196] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/5e906c95388b444ba44ad99faf7ae821
I20260924 03:15:47.178118 20399 ts_tablet_manager.cc:1939] T 5e906c95388b444ba44ad99faf7ae821 P 506eda2d32c0411dbe0ec244f7bec1bc: Deleting consensus metadata
I20260924 03:15:47.179591 19897 catalog_manager.cc:4841] TS 506eda2d32c0411dbe0ec244f7bec1bc (127.14.227.131:39207): tablet 5e906c95388b444ba44ad99faf7ae821 (table test-table [id=a447159c520e4546a21ef355541d72c7]) successfully deleted
I20260924 03:15:47.283707 20400 tablet_replica.cc:330] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21: stopping tablet replica
I20260924 03:15:47.284195 20400 raft_consensus.cc:2229] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:47.284709 20400 raft_consensus.cc:2258] T 5e906c95388b444ba44ad99faf7ae821 P 21f1f2cb618d4b24aaaa3d8bb56e3a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:47.285112 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 19971
W20260924 03:15:47.308122 19899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.227.129:46735 (error 108)
I20260924 03:15:47.308894 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 20102
I20260924 03:15:47.310184 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.309627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51636)
0924 03:15:47.309808 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.310008 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 03:15:47.310390 19899 catalog_manager.cc:4584] TS 21f1f2cb618d4b24aaaa3d8bb56e3a21 (127.14.227.129:46735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e906c95388b444ba44ad99faf7ae821: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
I20260924 03:15:47.312162 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.309990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.14.227.130:37039)
0924 03:15:47.310156 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:47.310163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:47.310218 (+ 55us) server_negotiation.cc:407] Connection header received
0924 03:15:47.310464 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:47.310470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:47.310576 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:47.310676 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:47.311542 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:47.311549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:47.311554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:47.311571 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:47.311609 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:47.311625 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:47.311757 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:47.311814 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 03:15:47.311870 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:15:47.312919 20224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:47.363933 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.363457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51652)
0924 03:15:47.363697 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.363782 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 03:15:47.365067 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.364702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51664)
0924 03:15:47.364889 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.364961 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 03:15:47.424316 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 20102
I20260924 03:15:47.427229 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.426752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51674)
0924 03:15:47.426945 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.427055 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 03:15:47.428448 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.427984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51678)
0924 03:15:47.428207 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.428291 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 03:15:47.441395 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 20232
I20260924 03:15:47.444445 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.442419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.14.227.131:39207)
0924 03:15:47.442585 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:47.442590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:47.442607 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:15:47.442681 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:47.442684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:47.442728 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:47.442806 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:47.443757 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:47.443763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:47.443767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:47.443790 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:47.443818 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:47.443829 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:47.443931 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:47.444069 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 03:15:47.444115 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:15:47.445102 20354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:47.509629 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.509140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51694)
0924 03:15:47.509349 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.509440 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 03:15:47.510782 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.510405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51700)
0924 03:15:47.510606 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.510671 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 03:15:47.563232 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 20232
I20260924 03:15:47.580945 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 19881
I20260924 03:15:47.582089 19942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:47.686165 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:47.685375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46735 (local address 127.0.0.1:51708)
0924 03:15:47.685805 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:47.685933 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260924 03:15:47.686182 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 19881
2026-09-24T03:15:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2652 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 03:15:47.727093 15246 mini_oidc.cc:152] Starting JWKS server
I20260924 03:15:47.729633 15246 webserver.cc:460] Webserver started at https://127.0.0.1:46605/ using document root <none> and password file <none>
I20260924 03:15:47.729938 15246 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:15:47.730599 15246 webserver.cc:460] Webserver started at http://127.0.0.1:39313/ using document root <none> and password file <none>
2026-09-24T03:15:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:47Z Disabled control of system clock
I20260924 03:15:47.755196 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:42327
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:42327
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46605/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/testchainca.cert with env {}
W20260924 03:15:48.075173 20413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:48.075464 20413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:48.075563 20413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:48.082906 20413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:48.083021 20413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:48.083062 20413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:48.083127 20413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:48.083175 20413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:15:48.083238 20413 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:15:48.083283 20413 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46605/jwks.json
I20260924 03:15:48.093062 20413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:42327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:42327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46605/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:48.094291 20413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:48.095614 20413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:48.107643 20419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:48.107894 20418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:48.107671 20421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:48.108403 20413 server_base.cc:1034] running on GCE node
I20260924 03:15:48.109297 20413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:48.110671 20413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:48.111883 20413 hybrid_clock.cc:648] HybridClock initialized: now 1790219748111848 us; error 47 us; skew 500 ppm
I20260924 03:15:48.112212 20413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:15:48.114420 20413 webserver.cc:460] Webserver started at http://127.14.227.190:35731/ using document root <none> and password file <none>
I20260924 03:15:48.115051 20413 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:48.115185 20413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:48.115485 20413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:48.118273 20413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "649b6924fb9443f2bab37ec5a3b9d4e4"
format_stamp: "Formatted at 2026-09-24 03:15:48 on dist-test-slave-vd0p"
server_key: "8cd8ec8b214e1680dc4008ddc9cda791"
server_key_iv: "f0361e107ac074bf7884ce8e3a85b91d"
server_key_version: "encryptionkey@0"
I20260924 03:15:48.119040 20413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "649b6924fb9443f2bab37ec5a3b9d4e4"
format_stamp: "Formatted at 2026-09-24 03:15:48 on dist-test-slave-vd0p"
server_key: "8cd8ec8b214e1680dc4008ddc9cda791"
server_key_iv: "f0361e107ac074bf7884ce8e3a85b91d"
server_key_version: "encryptionkey@0"
I20260924 03:15:48.123904 20413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:15:48.127197 20427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:48.128604 20413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:15:48.128733 20413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "649b6924fb9443f2bab37ec5a3b9d4e4"
format_stamp: "Formatted at 2026-09-24 03:15:48 on dist-test-slave-vd0p"
server_key: "8cd8ec8b214e1680dc4008ddc9cda791"
server_key_iv: "f0361e107ac074bf7884ce8e3a85b91d"
server_key_version: "encryptionkey@0"
I20260924 03:15:48.128894 20413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:48.155799 20413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:48.156528 20413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:48.156771 20413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:48.174258 20413 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:42327
I20260924 03:15:48.174276 20478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:42327 every 8 connection(s)
I20260924 03:15:48.175560 20413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:48.179124 20479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:48.181293 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 20413
I20260924 03:15:48.181517 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:48.182163 15246 external_mini_cluster.cc:1428] Setting key a6f2c6a10b643caaf66a22f7e3e78dbb
I20260924 03:15:48.191898 20479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4: Bootstrap starting.
I20260924 03:15:48.195699 20479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:48.197028 20479 log.cc:824] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:48.197706 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:48.184091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.14.227.190:42327)
0924 03:15:48.185128 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:48.185143 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:48.185190 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:15:48.187447 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:48.187473 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:48.188057 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:48.188475 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:48.189016 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:48.190998 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:48.192494 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:48.193561 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:48.193753 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:48.194871 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:48.194881 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:48.194897 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:48.195564 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:48.195694 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:48.195722 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:48.195840 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:48.196065 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 03:15:48.196394 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":328,"threads_started":1}
I20260924 03:15:48.199913 20479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4: No bootstrap required, opened a new log
I20260924 03:15:48.201414 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:15:48.207729 20479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649b6924fb9443f2bab37ec5a3b9d4e4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42327 } }
I20260924 03:15:48.208086 20479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:48.208185 20479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b6924fb9443f2bab37ec5a3b9d4e4, State: Initialized, Role: FOLLOWER
I20260924 03:15:48.208738 20479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649b6924fb9443f2bab37ec5a3b9d4e4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42327 } }
I20260924 03:15:48.209002 20479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:48.209133 20479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:48.209295 20479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:48.211930 20479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649b6924fb9443f2bab37ec5a3b9d4e4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42327 } }
I20260924 03:15:48.212491 20479 leader_election.cc:302] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 649b6924fb9443f2bab37ec5a3b9d4e4; no voters:
I20260924 03:15:48.213040 20479 leader_election.cc:288] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:48.213200 20484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:48.214272 20484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [term 1 LEADER]: Becoming Leader. State: Replica: 649b6924fb9443f2bab37ec5a3b9d4e4, State: Running, Role: LEADER
I20260924 03:15:48.214748 20484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649b6924fb9443f2bab37ec5a3b9d4e4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42327 } }
I20260924 03:15:48.215062 20479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:48.217824 20485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "649b6924fb9443f2bab37ec5a3b9d4e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649b6924fb9443f2bab37ec5a3b9d4e4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42327 } } }
I20260924 03:15:48.218308 20485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:48.218871 20486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 649b6924fb9443f2bab37ec5a3b9d4e4. Latest consensus state: current_term: 1 leader_uuid: "649b6924fb9443f2bab37ec5a3b9d4e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649b6924fb9443f2bab37ec5a3b9d4e4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42327 } } }
I20260924 03:15:48.219238 20486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:48.219956 20491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:48.224454 20491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:48.232389 20491 catalog_manager.cc:1347] Generated new cluster ID: 3d989daa47ce4bcf835952d6abc98564
I20260924 03:15:48.232503 20491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:48.278882 20491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:48.280640 20491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:48.291930 20491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 649b6924fb9443f2bab37ec5a3b9d4e4: Generated new TSK 0
I20260924 03:15:48.292939 20491 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:15:48.338766 20509 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 03:15:48.338922 20509 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:15:48.340019 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:48.317872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.14.227.190:42327)
0924 03:15:48.318077 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:48.318084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:48.318546 (+ 462us) server_negotiation.cc:407] Connection header received
0924 03:15:48.318834 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:48.318839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:48.318924 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:48.319077 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:15:48.319577 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:48.320704 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:48.322268 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:48.322954 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:48.323079 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:48.323552 (+ 473us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:15:48.339390 (+ 15838us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:15:48.339736 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60100: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 03:15:48.340143 20482 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60100: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 03:15:48.341315 20508 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:48.317797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:42327 (local address 127.0.0.1:60100)
0924 03:15:48.318451 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:48.318466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:48.318603 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:48.319116 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:48.319119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:48.319128 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:15:48.319434 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:48.319441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:48.320825 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:48.320829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:48.322134 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:48.322145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:48.323109 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:48.323113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:48.323225 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:48.323278 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:15:48.339658 (+ 16380us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 03:15:48.339884 (+ 226us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.227.190:42327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":336,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.227.190:42327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 03:15:48.342965 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 20413
I20260924 03:15:48.343986 20474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:48.438954 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 20413
2026-09-24T03:15:48Z chronyd exiting
I20260924 03:15:48.488860 15246 test_util.cc:175] -----------------------------------------------
I20260924 03:15:48.489029 15246 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790219707325830-15246-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (766 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 03:15:48.492736 15246 mini_oidc.cc:152] Starting JWKS server
I20260924 03:15:48.494604 15246 webserver.cc:460] Webserver started at https://127.0.0.1:42491/ using document root <none> and password file <none>
I20260924 03:15:48.494849 15246 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:15:48.495486 15246 webserver.cc:460] Webserver started at http://127.0.0.1:39209/ using document root <none> and password file <none>
2026-09-24T03:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:48Z Disabled control of system clock
I20260924 03:15:48.519867 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37611
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:37611
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/testchainca.cert with env {}
W20260924 03:15:48.830829 20516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:48.831085 20516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:48.831257 20516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:48.838440 20516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:48.838546 20516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:48.838660 20516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:48.838760 20516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:48.838842 20516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:15:48.838927 20516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:15:48.838979 20516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42491/jwks.json
I20260924 03:15:48.848716 20516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:37611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:48.849936 20516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:48.851215 20516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:48.862880 20521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:48.862979 20524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:48.862959 20522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:48.864010 20516 server_base.cc:1034] running on GCE node
I20260924 03:15:48.864779 20516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:48.866147 20516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:48.867358 20516 hybrid_clock.cc:648] HybridClock initialized: now 1790219748867312 us; error 28 us; skew 500 ppm
I20260924 03:15:48.867648 20516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:15:48.869733 20516 webserver.cc:460] Webserver started at http://127.14.227.190:35525/ using document root <none> and password file <none>
I20260924 03:15:48.870437 20516 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:48.870522 20516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:48.870779 20516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:48.873471 20516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "4be39b73c9c448d8889be20ea62b318d"
format_stamp: "Formatted at 2026-09-24 03:15:48 on dist-test-slave-vd0p"
server_key: "df3e8c7c9d89a3db7b7c0d047c5f9a0a"
server_key_iv: "dd88dd917fd3dc4ddf726776f92dbf0f"
server_key_version: "encryptionkey@0"
I20260924 03:15:48.874243 20516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "4be39b73c9c448d8889be20ea62b318d"
format_stamp: "Formatted at 2026-09-24 03:15:48 on dist-test-slave-vd0p"
server_key: "df3e8c7c9d89a3db7b7c0d047c5f9a0a"
server_key_iv: "dd88dd917fd3dc4ddf726776f92dbf0f"
server_key_version: "encryptionkey@0"
I20260924 03:15:48.879032 20516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:15:48.882103 20530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:48.883296 20516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 03:15:48.883435 20516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "4be39b73c9c448d8889be20ea62b318d"
format_stamp: "Formatted at 2026-09-24 03:15:48 on dist-test-slave-vd0p"
server_key: "df3e8c7c9d89a3db7b7c0d047c5f9a0a"
server_key_iv: "dd88dd917fd3dc4ddf726776f92dbf0f"
server_key_version: "encryptionkey@0"
I20260924 03:15:48.883600 20516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:48.909526 20516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:48.910348 20516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:48.910671 20516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:48.928303 20516 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:37611
I20260924 03:15:48.928309 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:37611 every 8 connection(s)
I20260924 03:15:48.929632 20516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:48.933301 20582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:48.935920 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 20516
I20260924 03:15:48.936094 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:48.936923 15246 external_mini_cluster.cc:1428] Setting key f514a656b7a389f15156272e5675b020
I20260924 03:15:48.945204 20582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d: Bootstrap starting.
I20260924 03:15:48.948621 20582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:48.949666 20582 log.cc:824] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:48.950100 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:48.938840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.14.227.190:37611)
0924 03:15:48.939813 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:48.939824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:48.939862 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:15:48.941433 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:48.941454 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:48.941939 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:48.942323 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:48.942876 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:48.944075 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:48.945875 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:48.946568 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:48.946707 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:48.947861 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:48.947868 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:48.947880 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:48.948352 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:48.948430 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:48.948448 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:48.948541 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:48.948793 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 03:15:48.949118 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":401,"threads_started":1}
I20260924 03:15:48.952497 20582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d: No bootstrap required, opened a new log
I20260924 03:15:48.953677 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:15:48.960209 20582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be39b73c9c448d8889be20ea62b318d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37611 } }
I20260924 03:15:48.960561 20582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:48.960664 20582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be39b73c9c448d8889be20ea62b318d, State: Initialized, Role: FOLLOWER
I20260924 03:15:48.961205 20582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be39b73c9c448d8889be20ea62b318d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37611 } }
I20260924 03:15:48.961422 20582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:48.961560 20582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:48.961722 20582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:48.964134 20582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be39b73c9c448d8889be20ea62b318d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37611 } }
I20260924 03:15:48.964609 20582 leader_election.cc:302] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4be39b73c9c448d8889be20ea62b318d; no voters:
I20260924 03:15:48.965152 20582 leader_election.cc:288] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:48.965328 20587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:48.966142 20587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [term 1 LEADER]: Becoming Leader. State: Replica: 4be39b73c9c448d8889be20ea62b318d, State: Running, Role: LEADER
I20260924 03:15:48.966709 20587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be39b73c9c448d8889be20ea62b318d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37611 } }
I20260924 03:15:48.967109 20582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:48.969146 20588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4be39b73c9c448d8889be20ea62b318d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be39b73c9c448d8889be20ea62b318d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37611 } } }
I20260924 03:15:48.969482 20588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:48.969998 20589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be39b73c9c448d8889be20ea62b318d. Latest consensus state: current_term: 1 leader_uuid: "4be39b73c9c448d8889be20ea62b318d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be39b73c9c448d8889be20ea62b318d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37611 } } }
I20260924 03:15:48.970327 20589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:48.971879 20593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:48.978309 20593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:48.986099 20593 catalog_manager.cc:1347] Generated new cluster ID: d2afce40608d4476afddb193668a25ce
I20260924 03:15:48.986202 20593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:48.998921 20593 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:49.000370 20593 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:49.015769 20593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4be39b73c9c448d8889be20ea62b318d: Generated new TSK 0
I20260924 03:15:49.016855 20593 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 03:15:49.050676 20613 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 03:15:49.050806 20613 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 03:15:49.052029 20612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:49.032352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37611 (local address 127.0.0.1:54484)
0924 03:15:49.032983 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:49.033002 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:49.033113 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:49.033555 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:49.033558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:49.033566 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:15:49.033789 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:49.033796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.035266 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.035270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:49.036313 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:49.036323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.037360 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.037364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:49.037494 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:49.037523 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:15:49.051536 (+ 14013us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:15:49.051730 (+ 194us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.227.190:37611: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":354,"threads_started":1}
I20260924 03:15:49.052018 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:49.032524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.14.227.190:37611)
0924 03:15:49.032721 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:49.032730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:49.033086 (+ 356us) server_negotiation.cc:407] Connection header received
0924 03:15:49.033271 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:49.033277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:49.033367 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:49.033546 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:15:49.033968 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.035136 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.036452 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.037212 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.037355 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:49.037801 (+ 446us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:15:49.051328 (+ 13527us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 03:15:49.051703 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54484: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 03:15:49.052138 20585 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54484: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 03:15:49.053364 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 20516
I20260924 03:15:49.054518 20577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:49.146251 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 20516
2026-09-24T03:15:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (707 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 03:15:49.200472 15246 mini_oidc.cc:152] Starting JWKS server
I20260924 03:15:49.202410 15246 webserver.cc:460] Webserver started at https://127.0.0.1:42743/ using document root <none> and password file <none>
I20260924 03:15:49.202646 15246 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 03:15:49.203291 15246 webserver.cc:460] Webserver started at http://127.0.0.1:33545/ using document root <none> and password file <none>
2026-09-24T03:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:49Z Disabled control of system clock
I20260924 03:15:49.228302 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44917
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:44917
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42743/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:15:49.545392 20620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:49.545740 20620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:49.545910 20620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:49.553259 20620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:49.553395 20620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:49.553485 20620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:49.553591 20620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:49.553675 20620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:15:49.553771 20620 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:15:49.553848 20620 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42743/jwks.json
I20260924 03:15:49.563156 20620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:44917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42743/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:49.564455 20620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:49.566294 20620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:49.577939 20625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:49.577939 20626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:49.577980 20628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:49.578979 20620 server_base.cc:1034] running on GCE node
I20260924 03:15:49.579761 20620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:49.581041 20620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:49.582255 20620 hybrid_clock.cc:648] HybridClock initialized: now 1790219749582220 us; error 52 us; skew 500 ppm
I20260924 03:15:49.582587 20620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:15:49.589815 20620 webserver.cc:460] Webserver started at http://127.14.227.190:34533/ using document root <none> and password file <none>
I20260924 03:15:49.590546 20620 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:49.590672 20620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:49.590981 20620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:49.593832 20620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4"
format_stamp: "Formatted at 2026-09-24 03:15:49 on dist-test-slave-vd0p"
server_key: "19ec06c29a9585a6cfa281310fc46908"
server_key_iv: "11f6700badfc8c5330a6515b054b6635"
server_key_version: "encryptionkey@0"
I20260924 03:15:49.594660 20620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4"
format_stamp: "Formatted at 2026-09-24 03:15:49 on dist-test-slave-vd0p"
server_key: "19ec06c29a9585a6cfa281310fc46908"
server_key_iv: "11f6700badfc8c5330a6515b054b6635"
server_key_version: "encryptionkey@0"
I20260924 03:15:49.599522 20620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:15:49.602686 20634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:49.604027 20620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:49.604194 20620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4"
format_stamp: "Formatted at 2026-09-24 03:15:49 on dist-test-slave-vd0p"
server_key: "19ec06c29a9585a6cfa281310fc46908"
server_key_iv: "11f6700badfc8c5330a6515b054b6635"
server_key_version: "encryptionkey@0"
I20260924 03:15:49.604416 20620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:49.617012 20620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:49.617837 20620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:49.618172 20620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:49.635667 20620 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:44917
I20260924 03:15:49.635685 20685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:44917 every 8 connection(s)
I20260924 03:15:49.637006 20620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:49.640630 20686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:49.644618 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 20620
I20260924 03:15:49.644824 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:49.645634 15246 external_mini_cluster.cc:1428] Setting key 33c62ce8b0bfaf8ce588ab1b25ee4322
I20260924 03:15:49.651520 20686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4: Bootstrap starting.
I20260924 03:15:49.654373 20686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:49.655341 20686 log.cc:824] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:49.658614 20686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4: No bootstrap required, opened a new log
I20260924 03:15:49.658958 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:49.647530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.14.227.190:44917)
0924 03:15:49.648396 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:49.648406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:49.648449 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:15:49.650314 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:49.650334 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:49.650751 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:49.651125 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:49.651708 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.652952 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.654212 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.654938 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.655126 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:49.656372 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:49.656379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:49.656392 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:49.657008 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:49.657100 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:49.657128 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:49.657259 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:49.657463 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:15:49.657768 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":309,"threads_started":1}
I20260924 03:15:49.662544 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:15:49.666596 20686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44917 } }
I20260924 03:15:49.666890 20686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:49.666960 20686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cedd08862bc41b5a95e6f1b5f9b6ee4, State: Initialized, Role: FOLLOWER
I20260924 03:15:49.667429 20686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44917 } }
I20260924 03:15:49.667610 20686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:49.667698 20686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:49.667829 20686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:49.670091 20686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44917 } }
I20260924 03:15:49.670503 20686 leader_election.cc:302] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cedd08862bc41b5a95e6f1b5f9b6ee4; no voters:
I20260924 03:15:49.670998 20686 leader_election.cc:288] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:49.671129 20691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:49.671841 20691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cedd08862bc41b5a95e6f1b5f9b6ee4, State: Running, Role: LEADER
I20260924 03:15:49.672329 20691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44917 } }
I20260924 03:15:49.672981 20686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:49.675066 20692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44917 } } }
I20260924 03:15:49.675379 20692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:49.676888 20698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:49.676949 20693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cedd08862bc41b5a95e6f1b5f9b6ee4. Latest consensus state: current_term: 1 leader_uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cedd08862bc41b5a95e6f1b5f9b6ee4" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44917 } } }
I20260924 03:15:49.677235 20693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:49.681715 20698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:49.689586 20698 catalog_manager.cc:1347] Generated new cluster ID: 6fb71445048440cbbbc197870f156c01
I20260924 03:15:49.689709 20698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:49.709350 20698 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:49.711179 20698 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:49.720237 20698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cedd08862bc41b5a95e6f1b5f9b6ee4: Generated new TSK 0
I20260924 03:15:49.721141 20698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:15:49.755334 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:49.746982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.14.227.190:44917)
0924 03:15:49.747177 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:49.747184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:49.747680 (+ 496us) server_negotiation.cc:407] Connection header received
0924 03:15:49.747900 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:49.747907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:49.747996 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:49.748157 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:15:49.748639 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.749708 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.751060 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.751738 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.751860 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:49.752309 (+ 449us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:15:49.754721 (+ 2412us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:15:49.755042 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 03:15:49.755460 20689 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 03:15:49.757016 20715 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:49.746878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44917 (local address 127.0.0.1:55084)
0924 03:15:49.747576 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:49.747599 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:49.747722 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:49.748192 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:49.748196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:49.748206 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:15:49.748474 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:49.748481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.749840 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.749843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:49.750926 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:49.750936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:49.751911 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:49.751915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:49.752035 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:49.752064 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:15:49.754919 (+ 2855us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 03:15:49.756608 (+ 1689us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.227.190:44917: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":1419,"thread_start_us":379,"threads_started":1}
I20260924 03:15:49.758037 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 20620
I20260924 03:15:49.759269 20681 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 03:15:49.770339 20716 webserver.cc:533] Webserver: sslize failed:
E20260924 03:15:49.770457 20716 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 03:15:49.851918 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 20620
2026-09-24T03:15:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (705 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T03:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:49Z Disabled control of system clock
I20260924 03:15:49.930496 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37837
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:37837
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/testchainca.cert with env {}
W20260924 03:15:50.238600 20721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:50.238893 20721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:50.238981 20721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:50.246090 20721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:50.246196 20721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:50.246235 20721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:50.246284 20721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:50.246320 20721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 03:15:50.246371 20721 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 03:15:50.246410 20721 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 03:15:50.255977 20721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:37837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:50.257164 20721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:50.258388 20721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:50.270035 20726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:50.269980 20727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:50.270102 20729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:50.270689 20721 server_base.cc:1034] running on GCE node
I20260924 03:15:50.271498 20721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:50.272869 20721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:50.274081 20721 hybrid_clock.cc:648] HybridClock initialized: now 1790219750274031 us; error 28 us; skew 500 ppm
I20260924 03:15:50.274441 20721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:15:50.276717 20721 webserver.cc:460] Webserver started at http://127.14.227.190:33295/ using document root <none> and password file <none>
I20260924 03:15:50.277364 20721 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:50.277489 20721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:50.277782 20721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:50.280597 20721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "c8e3090f52354aeeae8e3b59db44addd"
format_stamp: "Formatted at 2026-09-24 03:15:50 on dist-test-slave-vd0p"
server_key: "87eeb5862c8cbba862d80d9b2d3f5000"
server_key_iv: "6be8b23c3f4332c593d189d8fc33455f"
server_key_version: "encryptionkey@0"
I20260924 03:15:50.281364 20721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "c8e3090f52354aeeae8e3b59db44addd"
format_stamp: "Formatted at 2026-09-24 03:15:50 on dist-test-slave-vd0p"
server_key: "87eeb5862c8cbba862d80d9b2d3f5000"
server_key_iv: "6be8b23c3f4332c593d189d8fc33455f"
server_key_version: "encryptionkey@0"
I20260924 03:15:50.286588 20721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:15:50.290237 20735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:50.291563 20721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:15:50.291759 20721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "c8e3090f52354aeeae8e3b59db44addd"
format_stamp: "Formatted at 2026-09-24 03:15:50 on dist-test-slave-vd0p"
server_key: "87eeb5862c8cbba862d80d9b2d3f5000"
server_key_iv: "6be8b23c3f4332c593d189d8fc33455f"
server_key_version: "encryptionkey@0"
I20260924 03:15:50.291980 20721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:50.311448 20721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:50.312220 20721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:50.312498 20721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:50.330204 20721 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:37837
I20260924 03:15:50.330180 20786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:37837 every 8 connection(s)
I20260924 03:15:50.331414 20721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:50.334826 20787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:50.337213 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 20721
I20260924 03:15:50.337374 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:50.338141 15246 external_mini_cluster.cc:1428] Setting key adc49fac06a6918248f227b107157a2a
I20260924 03:15:50.346463 20787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd: Bootstrap starting.
I20260924 03:15:50.349895 20787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:50.350947 20787 log.cc:824] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:50.351307 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:50.340061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.14.227.190:37837)
0924 03:15:50.341014 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:50.341024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:50.341061 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:15:50.342650 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:50.342670 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:50.343094 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:50.343425 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:50.343981 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:50.345242 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:50.346982 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:50.347636 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:50.347754 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:50.348990 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:50.348996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:50.349008 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:15:50.349502 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:50.349594 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:15:50.349613 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:50.349696 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:50.350002 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 03:15:50.350362 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260924 03:15:50.354041 20787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd: No bootstrap required, opened a new log
I20260924 03:15:50.355278 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:15:50.362260 20787 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e3090f52354aeeae8e3b59db44addd" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37837 } }
I20260924 03:15:50.362548 20787 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:50.362613 20787 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e3090f52354aeeae8e3b59db44addd, State: Initialized, Role: FOLLOWER
I20260924 03:15:50.363090 20787 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e3090f52354aeeae8e3b59db44addd" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37837 } }
I20260924 03:15:50.363268 20787 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:50.363359 20787 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:50.363538 20787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:50.365775 20787 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e3090f52354aeeae8e3b59db44addd" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37837 } }
I20260924 03:15:50.366289 20787 leader_election.cc:302] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e3090f52354aeeae8e3b59db44addd; no voters:
I20260924 03:15:50.366832 20787 leader_election.cc:288] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:50.366971 20792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:50.367910 20792 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [term 1 LEADER]: Becoming Leader. State: Replica: c8e3090f52354aeeae8e3b59db44addd, State: Running, Role: LEADER
I20260924 03:15:50.368570 20792 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e3090f52354aeeae8e3b59db44addd" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37837 } }
I20260924 03:15:50.368973 20787 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:50.371145 20793 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e3090f52354aeeae8e3b59db44addd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e3090f52354aeeae8e3b59db44addd" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37837 } } }
I20260924 03:15:50.371212 20794 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e3090f52354aeeae8e3b59db44addd. Latest consensus state: current_term: 1 leader_uuid: "c8e3090f52354aeeae8e3b59db44addd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e3090f52354aeeae8e3b59db44addd" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37837 } } }
I20260924 03:15:50.371515 20793 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:50.371512 20794 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:50.373327 20801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:50.378041 20801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:50.386197 20801 catalog_manager.cc:1347] Generated new cluster ID: 51b2004371f74a6a8eb90383070884c2
I20260924 03:15:50.386298 20801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:50.401909 20801 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:50.403607 20801 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:50.417517 20801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e3090f52354aeeae8e3b59db44addd: Generated new TSK 0
I20260924 03:15:50.418552 20801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:15:50.471856 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:50.439120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.14.227.190:37837)
0924 03:15:50.439359 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:50.439370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:50.439893 (+ 523us) server_negotiation.cc:407] Connection header received
0924 03:15:50.440147 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:50.440151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:50.440220 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:50.440374 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0924 03:15:50.440818 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:50.441521 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:50.442972 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:50.443434 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:50.443589 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:50.444084 (+ 495us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 03:15:50.471136 (+ 27052us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:15:50.471528 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58564: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 03:15:50.472015 20790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58564: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 03:15:50.472095 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:50.439020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37837 (local address 127.0.0.1:58564)
0924 03:15:50.439796 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:50.439811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:50.439968 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:50.440384 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:50.440389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:50.440400 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 03:15:50.440673 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:50.440682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:50.441674 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:50.441681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:50.442820 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:50.442830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:50.443635 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:50.443642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:50.443788 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:50.443824 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 03:15:50.471403 (+ 27579us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 03:15:50.471686 (+ 283us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.227.190:37837: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":516,"threads_started":1}
I20260924 03:15:50.473202 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 20721
I20260924 03:15:50.474633 20782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:50.567812 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 20721
2026-09-24T03:15:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (692 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (393 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (394 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:51 dist-test-slave-vd0p krb5kdc[20831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:51 dist-test-slave-vd0p krb5kdc[20831](info): set up 2 sockets
Sep 24 03:15:51 dist-test-slave-vd0p krb5kdc[20831](info): commencing operation
krb5kdc: starting...
W20260924 03:15:52.414525 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:52 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:15:52.604773 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44907
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:52.914875 20844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:52.915205 20844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:52.915380 20844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:52.922534 20844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:52.922669 20844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:52.922773 20844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:52.922881 20844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:52.922986 20844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:52.932592 20844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:52.933897 20844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:52.935199 20844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:52.946437 20849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:52.946483 20850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:52.946506 20852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:52.947511 20844 server_base.cc:1034] running on GCE node
I20260924 03:15:52.948237 20844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:52.949532 20844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:52.950757 20844 hybrid_clock.cc:648] HybridClock initialized: now 1790219752950708 us; error 55 us; skew 500 ppm
I20260924 03:15:52.951090 20844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:52 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219752, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:52.953918 20844 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:15:52.955229 20844 webserver.cc:460] Webserver started at http://127.14.227.190:44805/ using document root <none> and password file <none>
I20260924 03:15:52.955813 20844 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:52.955902 20844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:52.956130 20844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:52.958937 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "3d8953d67e5340068a88763a95753bc8"
format_stamp: "Formatted at 2026-09-24 03:15:52 on dist-test-slave-vd0p"
server_key: "404d6ba3b0089165ed63c439c50343bf"
server_key_iv: "ff4477fbfdbdf46d00ed95d9ee6ce325"
server_key_version: "encryptionkey@0"
I20260924 03:15:52.959650 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "3d8953d67e5340068a88763a95753bc8"
format_stamp: "Formatted at 2026-09-24 03:15:52 on dist-test-slave-vd0p"
server_key: "404d6ba3b0089165ed63c439c50343bf"
server_key_iv: "ff4477fbfdbdf46d00ed95d9ee6ce325"
server_key_version: "encryptionkey@0"
I20260924 03:15:52.964762 20844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:15:52.967833 20859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:52.969036 20844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:15:52.969180 20844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "3d8953d67e5340068a88763a95753bc8"
format_stamp: "Formatted at 2026-09-24 03:15:52 on dist-test-slave-vd0p"
server_key: "404d6ba3b0089165ed63c439c50343bf"
server_key_iv: "ff4477fbfdbdf46d00ed95d9ee6ce325"
server_key_version: "encryptionkey@0"
I20260924 03:15:52.969350 20844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:52.987108 20844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:52.990852 20844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:52.991070 20844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:53.008876 20844 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:44907
I20260924 03:15:53.008898 20910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:44907 every 8 connection(s)
I20260924 03:15:53.010511 20844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:53.011910 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 20844
I20260924 03:15:53.012104 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:53.012735 15246 external_mini_cluster.cc:1428] Setting key 6a6741899a22bb4fc749ee13ef296995
I20260924 03:15:53.015110 20911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:15:53 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:53.027815 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8: Bootstrap starting.
I20260924 03:15:53.031838 20911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:53.033042 20911 log.cc:824] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:53.036330 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8: No bootstrap required, opened a new log
I20260924 03:15:53.036609 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:53.014761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.14.227.190:44907)
0924 03:15:53.015572 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:53.015588 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:53.015637 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:15:53.017912 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:53.017939 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:53.018470 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:53.018902 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:53.020230 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:53.022215 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:53.023435 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:53.024463 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:53.024646 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:53.028462 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:53.028472 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:53.028490 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:53.028536 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:53.032396 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:53.033014 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:53.033025 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:53.033035 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:53.033235 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:53.033695 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:53.033700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:53.033703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:53.034319 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:53.034605 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:53.034948 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:15:53.035362 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":365,"threads_started":1}
I20260924 03:15:53.043948 20911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8953d67e5340068a88763a95753bc8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44907 } }
I20260924 03:15:53.044251 20911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:53.044322 20911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8953d67e5340068a88763a95753bc8, State: Initialized, Role: FOLLOWER
I20260924 03:15:53.044836 20911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8953d67e5340068a88763a95753bc8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44907 } }
I20260924 03:15:53.045070 20911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:53.045228 20911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:53.045424 20911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:53.047564 20911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8953d67e5340068a88763a95753bc8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44907 } }
I20260924 03:15:53.047996 20911 leader_election.cc:302] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8953d67e5340068a88763a95753bc8; no voters:
I20260924 03:15:53.048512 20911 leader_election.cc:288] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:53.048669 20916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:53.049391 20916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 3d8953d67e5340068a88763a95753bc8, State: Running, Role: LEADER
I20260924 03:15:53.049893 20916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8953d67e5340068a88763a95753bc8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44907 } }
I20260924 03:15:53.050705 20911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:53.052780 20917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d8953d67e5340068a88763a95753bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8953d67e5340068a88763a95753bc8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44907 } } }
I20260924 03:15:53.053367 20917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:53.053099 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8953d67e5340068a88763a95753bc8. Latest consensus state: current_term: 1 leader_uuid: "3d8953d67e5340068a88763a95753bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8953d67e5340068a88763a95753bc8" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44907 } } }
I20260924 03:15:53.053649 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:53.061565 20924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:53.066128 20924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:53.074554 20924 catalog_manager.cc:1347] Generated new cluster ID: d570f9ae6b714758b13c9023965b8948
I20260924 03:15:53.074672 20924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:53.083767 20924 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:53.085525 20924 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:53.096933 20924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d8953d67e5340068a88763a95753bc8: Generated new TSK 0
I20260924 03:15:53.098121 20924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:15:53.151068 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44907
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:53.478205 20937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:53.478761 20937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:53.478936 20937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:53.486334 20937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:53.486447 20937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:53.486650 20937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:53.496411 20937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:53.497763 20937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:53.499143 20937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:53.512055 20942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:53.512194 20945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:53.512113 20943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:53.512959 20937 server_base.cc:1034] running on GCE node
I20260924 03:15:53.513511 20937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:53.514509 20937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:53.515722 20937 hybrid_clock.cc:648] HybridClock initialized: now 1790219753515674 us; error 36 us; skew 500 ppm
I20260924 03:15:53.516023 20937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:53 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:53.519126 20937 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:15:53.520643 20937 webserver.cc:460] Webserver started at http://127.14.227.129:33493/ using document root <none> and password file <none>
I20260924 03:15:53.521301 20937 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:53.521430 20937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:53.521734 20937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:53.524561 20937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "ed18a271e993472b956b40b58929d4ea"
format_stamp: "Formatted at 2026-09-24 03:15:53 on dist-test-slave-vd0p"
server_key: "73f3f9887ae9708e90001577df7cb029"
server_key_iv: "3225ec5f740471d362722648ec7f8bcd"
server_key_version: "encryptionkey@0"
I20260924 03:15:53.525319 20937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "ed18a271e993472b956b40b58929d4ea"
format_stamp: "Formatted at 2026-09-24 03:15:53 on dist-test-slave-vd0p"
server_key: "73f3f9887ae9708e90001577df7cb029"
server_key_iv: "3225ec5f740471d362722648ec7f8bcd"
server_key_version: "encryptionkey@0"
I20260924 03:15:53.530213 20937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 03:15:53.533370 20952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:53.534658 20937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:53.534837 20937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "ed18a271e993472b956b40b58929d4ea"
format_stamp: "Formatted at 2026-09-24 03:15:53 on dist-test-slave-vd0p"
server_key: "73f3f9887ae9708e90001577df7cb029"
server_key_iv: "3225ec5f740471d362722648ec7f8bcd"
server_key_version: "encryptionkey@0"
I20260924 03:15:53.535043 20937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:53.556568 20937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:53.560381 20937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:53.560678 20937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:53.561698 20937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:53.563194 20937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:53.563292 20937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:53.563422 20937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:53.563493 20937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:53.595605 20937 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:38547
I20260924 03:15:53.595691 21064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:38547 every 8 connection(s)
I20260924 03:15:53.596954 20937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:53.602025 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 20937
I20260924 03:15:53.602223 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:53.602857 15246 external_mini_cluster.cc:1428] Setting key 59d9d3a250c35aa4ba2a3f5df5569a03
Sep 24 03:15:53 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:53.617532 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:53.599512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:43749 (local address 127.14.227.190:44907)
0924 03:15:53.600024 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:53.600029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:53.600991 (+ 962us) server_negotiation.cc:407] Connection header received
0924 03:15:53.603021 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:53.603027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:53.603114 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:53.603233 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:53.605195 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:53.606453 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:53.607988 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:53.608407 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:53.608543 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:53.612020 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:53.612025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:53.612028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:53.612058 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:53.614586 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:53.615285 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:53.615292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:53.615296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:53.615415 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:53.615839 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:53.615845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:53.615849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:53.616108 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:53.616263 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:53.616941 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 03:15:53.617149 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":318,"threads_started":1}
I20260924 03:15:53.618569 21068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:53.600149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44907 (local address 127.14.227.129:43749)
0924 03:15:53.600840 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:53.600877 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:53.602717 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:53.603498 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:53.603510 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:53.604250 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:53.605034 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:53.605054 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:53.606611 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:53.606623 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:53.607840 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:53.607851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:53.608634 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:53.608641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:53.608877 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:53.609560 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:53.609586 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:53.611801 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:53.614728 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:53.614737 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:53.614753 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:53.615162 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:53.615525 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:53.615530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:53.615534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:53.615729 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:53.616249 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:53.616257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:53.616589 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 03:15:53.616860 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 03:15:53.617387 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":268,"threads_started":1}
I20260924 03:15:53.620234 21065 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44907
I20260924 03:15:53.620527 21065 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:15:53.621321 21065 heartbeater.cc:511] Master 127.14.227.190:44907 requested a full tablet report, sending...
I20260924 03:15:53.623335 20876 ts_manager.cc:194] Registered new tserver with Master: ed18a271e993472b956b40b58929d4ea (127.14.227.129:38547)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:15:53.624826 20876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:43749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:15:53.646890 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44907
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:53.954361 21071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:53.954659 21071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:53.954830 21071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:53.961478 21071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:53.961616 21071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:53.961781 21071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:53.970825 21071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:53.972018 21071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:53.973387 21071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:53.985996 21076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:53.985940 21079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:53.986043 21077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:53.991603 21071 server_base.cc:1034] running on GCE node
I20260924 03:15:53.992148 21071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:53.992972 21071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:53.994170 21071 hybrid_clock.cc:648] HybridClock initialized: now 1790219753994104 us; error 65 us; skew 500 ppm
I20260924 03:15:53.994493 21071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:53 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:53.997442 21071 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:15:53.998916 21071 webserver.cc:460] Webserver started at http://127.14.227.130:45435/ using document root <none> and password file <none>
I20260924 03:15:53.999596 21071 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:53.999747 21071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:54.000051 21071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:54.002923 21071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "6be69084de9d4c979bafb59ddc91a3af"
format_stamp: "Formatted at 2026-09-24 03:15:53 on dist-test-slave-vd0p"
server_key: "8acf1a10672f2afd0d06ad7edabcfac1"
server_key_iv: "43023ed60336eafb330416cc27f26ff2"
server_key_version: "encryptionkey@0"
I20260924 03:15:54.003675 21071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "6be69084de9d4c979bafb59ddc91a3af"
format_stamp: "Formatted at 2026-09-24 03:15:53 on dist-test-slave-vd0p"
server_key: "8acf1a10672f2afd0d06ad7edabcfac1"
server_key_iv: "43023ed60336eafb330416cc27f26ff2"
server_key_version: "encryptionkey@0"
I20260924 03:15:54.008849 21071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:15:54.013013 21086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:54.014593 21071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:54.014743 21071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "6be69084de9d4c979bafb59ddc91a3af"
format_stamp: "Formatted at 2026-09-24 03:15:53 on dist-test-slave-vd0p"
server_key: "8acf1a10672f2afd0d06ad7edabcfac1"
server_key_iv: "43023ed60336eafb330416cc27f26ff2"
server_key_version: "encryptionkey@0"
I20260924 03:15:54.014947 21071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:54.042125 21071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:54.045853 21071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:54.046150 21071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:54.047012 21071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:54.048197 21071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:54.048267 21071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:54.048347 21071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:54.048391 21071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:54.080273 21071 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:41761
I20260924 03:15:54.080328 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:41761 every 8 connection(s)
I20260924 03:15:54.081486 21071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:54.084345 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 21071
I20260924 03:15:54.084502 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:54.084935 15246 external_mini_cluster.cc:1428] Setting key a0e5303a4d0500d7272c8754f096d0eb
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219753, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:54.103631 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.083926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:48751 (local address 127.14.227.190:44907)
0924 03:15:54.084133 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.084142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.085548 (+ 1406us) server_negotiation.cc:407] Connection header received
0924 03:15:54.087595 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.087611 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.087702 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.087806 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.089574 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.090699 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.092193 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.093004 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.093155 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.096905 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:54.096914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:54.096921 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:54.096963 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:54.100556 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.101201 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.101209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.101215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.101366 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.101731 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.101738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.101744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.102181 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:54.102330 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:54.102922 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 03:15:54.103186 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:15:54.104157 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.084527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44907 (local address 127.14.227.130:48751)
0924 03:15:54.085377 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:54.085427 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:54.087342 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:54.088079 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:54.088090 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:54.088893 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:54.089421 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:54.089436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.090834 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.090838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:54.092054 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:54.092065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.093239 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.093246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:54.093478 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.094160 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:54.094193 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:54.096705 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:54.100693 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:54.100699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:54.100711 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:54.101085 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:54.101470 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:54.101473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:54.101475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:54.101627 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:54.102302 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:54.102307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:54.102578 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 03:15:54.102786 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 03:15:54.103118 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":333,"threads_started":1}
I20260924 03:15:54.105578 21199 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44907
I20260924 03:15:54.105901 21199 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
I20260924 03:15:54.106729 21199 heartbeater.cc:511] Master 127.14.227.190:44907 requested a full tablet report, sending...
I20260924 03:15:54.108150 20876 ts_manager.cc:194] Registered new tserver with Master: 6be69084de9d4c979bafb59ddc91a3af (127.14.227.130:41761)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
I20260924 03:15:54.108985 20876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:48751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:15:54.132920 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44907
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:54.444530 21204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:54.444867 21204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:54.445043 21204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:54.452464 21204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:54.452617 21204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:54.452790 21204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:54.462577 21204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:54.463848 21204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:54.465164 21204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:54.477882 21209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:54.478024 21210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:54.478237 21212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:54.478997 21204 server_base.cc:1034] running on GCE node
I20260924 03:15:54.479614 21204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:54.480461 21204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:54.481657 21204 hybrid_clock.cc:648] HybridClock initialized: now 1790219754481608 us; error 38 us; skew 500 ppm
I20260924 03:15:54.482019 21204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:54.485036 21204 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:15:54.486469 21204 webserver.cc:460] Webserver started at http://127.14.227.131:35247/ using document root <none> and password file <none>
I20260924 03:15:54.487160 21204 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:54.487301 21204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:54.487605 21204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:54.490491 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "425aa97de18b47d8966dd8c160051a45"
format_stamp: "Formatted at 2026-09-24 03:15:54 on dist-test-slave-vd0p"
server_key: "b1a8c7f475d6e0401a6910fd33c875a9"
server_key_iv: "5632461a1c0410568efb78d42acb2939"
server_key_version: "encryptionkey@0"
I20260924 03:15:54.491251 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "425aa97de18b47d8966dd8c160051a45"
format_stamp: "Formatted at 2026-09-24 03:15:54 on dist-test-slave-vd0p"
server_key: "b1a8c7f475d6e0401a6910fd33c875a9"
server_key_iv: "5632461a1c0410568efb78d42acb2939"
server_key_version: "encryptionkey@0"
I20260924 03:15:54.496223 21204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:15:54.499392 21219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:54.500653 21204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:15:54.500819 21204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "425aa97de18b47d8966dd8c160051a45"
format_stamp: "Formatted at 2026-09-24 03:15:54 on dist-test-slave-vd0p"
server_key: "b1a8c7f475d6e0401a6910fd33c875a9"
server_key_iv: "5632461a1c0410568efb78d42acb2939"
server_key_version: "encryptionkey@0"
I20260924 03:15:54.501010 21204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:54.522568 21204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:54.526325 21204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:54.526611 21204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:54.527537 21204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:54.528789 21204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:54.528894 21204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:54.529039 21204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:54.529111 21204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:54.561152 21204 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:37035
I20260924 03:15:54.561204 21331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:37035 every 8 connection(s)
I20260924 03:15:54.562522 21204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:54.570920 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 21204
I20260924 03:15:54.571085 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:54.571616 15246 external_mini_cluster.cc:1428] Setting key 9b82edde5ffcca6a30433ad719e25f83
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:15:54.581480 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.564956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:42515 (local address 127.14.227.190:44907)
0924 03:15:54.565163 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.565170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.566395 (+ 1225us) server_negotiation.cc:407] Connection header received
0924 03:15:54.568095 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.568099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.568164 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.568261 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.569679 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.570480 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.572611 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.573120 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.573256 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.576742 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:54.576747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:54.576750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:54.576780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:54.579051 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.579623 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.579626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.579628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.579748 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.580068 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.580070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.580072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.580324 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:54.580424 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:54.580956 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 03:15:54.581137 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:15:54.582330 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.565491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44907 (local address 127.14.227.131:42515)
0924 03:15:54.566245 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:54.566281 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:54.567827 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:54.568441 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:54.568448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:54.568966 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:54.569512 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:54.569527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.570612 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.570617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:54.572446 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:54.572459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.573329 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.573337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:54.573584 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.574374 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:54.574406 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:54.576534 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:54.579162 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:54.579167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:54.579178 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:54.579532 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:54.579830 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:54.579834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:54.579836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:54.579985 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:54.580415 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:54.580420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:54.580686 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 03:15:54.580890 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 03:15:54.581229 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":272,"threads_started":1}
I20260924 03:15:54.583587 21332 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44907
I20260924 03:15:54.583891 21332 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:54.584571 21332 heartbeater.cc:511] Master 127.14.227.190:44907 requested a full tablet report, sending...
I20260924 03:15:54.586283 20876 ts_manager.cc:194] Registered new tserver with Master: 425aa97de18b47d8966dd8c160051a45 (127.14.227.131:37035)
I20260924 03:15:54.587085 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:54.587401 20876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:42515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/test-admin.keytab.
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:54.616003 15246 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 03:15:54.620909 21339 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:54.618384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.14.227.129:38547)
0924 03:15:54.619075 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.619083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.619111 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:15:54.619239 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.619246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.619478 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.619617 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.620402 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58772: BlockingRecv error: recv got EOF from 127.0.0.1:58772 (error 108)
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":308,"threads_started":1}
W20260924 03:15:54.620990 15246 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:54.622694 15246 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:54.624089 15246 init.cc:295] Successfully reacquired a new kerberos TGT
I20260924 03:15:54.628270 21065 heartbeater.cc:503] Master 127.14.227.190:44907 was elected leader, sending a full tablet report...
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:54.642524 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.625653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.14.227.129:38547)
0924 03:15:54.625796 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.625799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.626041 (+ 242us) server_negotiation.cc:407] Connection header received
0924 03:15:54.626253 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.626257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.626327 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.626407 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.627328 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.628501 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.629721 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.630488 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.630583 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.637098 (+ 6515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:54.637108 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:54.637122 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:54.637150 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:54.640136 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.640720 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.640726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.640728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.640843 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.641156 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.641159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.641161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.641465 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:54.641676 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:54.641931 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 03:15:54.642213 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 03:15:54.648108 21339 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:54.646354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.14.227.129:38547)
0924 03:15:54.646551 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.646560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.646818 (+ 258us) server_negotiation.cc:407] Connection header received
0924 03:15:54.647009 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.647014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.647085 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.647180 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.647856 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58796: BlockingRecv error: recv got EOF from 127.0.0.1:58796 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 03:15:54.648383 15246 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5cc' of type FILE
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:54.649748 15246 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:54.651114 15246 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:54.663812 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.652436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.14.227.129:38547)
0924 03:15:54.652598 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.652603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.652767 (+ 164us) server_negotiation.cc:407] Connection header received
0924 03:15:54.652937 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.652941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.653002 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.653103 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.653997 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.654671 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.655868 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.656253 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.656380 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.659499 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:54.659503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:54.659506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:54.659542 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:54.661766 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.662392 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.662400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.662406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.662523 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.662871 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.662874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.662876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.663105 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:54.663294 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:54.663457 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 03:15:54.663590 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 03:15:54.667651 21339 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:15:54.666184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.14.227.129:38547)
0924 03:15:54.666390 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.666395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.666568 (+ 173us) server_negotiation.cc:407] Connection header received
0924 03:15:54.666676 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.666679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.666739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.666824 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.667389 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58824: BlockingRecv error: recv got EOF from 127.0.0.1:58824 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 03:15:54.668012 15246 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790219707325830-15246-0/krb5kdc/krb5cc)
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:54.669378 15246 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:54.670619 15246 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:15:54.683586 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.671835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.14.227.129:38547)
0924 03:15:54.672005 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.672012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.672226 (+ 214us) server_negotiation.cc:407] Connection header received
0924 03:15:54.672396 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.672400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.672462 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.672563 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.673488 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.674259 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.675430 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.675832 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.675959 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.679049 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:54.679054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:54.679059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:54.679091 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:54.681190 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.681771 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.681776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.681779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.681904 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.682304 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.682311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.682314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.682573 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:54.682870 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:54.683112 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 03:15:54.683297 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:15:54.684975 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 20937
I20260924 03:15:54.695415 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.685747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.14.227.129:38547)
0924 03:15:54.685930 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.685935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.686111 (+ 176us) server_negotiation.cc:407] Connection header received
0924 03:15:54.686214 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.686217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.686261 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.686330 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.687144 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.687740 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.688822 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.689254 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.689345 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.691164 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:54.691168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:54.691171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:54.691194 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:54.693263 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.693789 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.693792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.693794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.693903 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.694303 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.694306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.694308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.694518 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:54.694712 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:54.694946 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 03:15:54.695134 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:15:54.695969 21060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:54.779060 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 20937
I20260924 03:15:54.795526 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 21071
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:15:54.810236 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.796412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.14.227.130:41761)
0924 03:15:54.796906 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.796913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.796938 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:15:54.796994 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.797000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.797213 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.797334 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.798297 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.799278 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.800380 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.800888 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.800999 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.805164 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:54.805173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:54.805188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:54.805223 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:54.807815 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.808413 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.808419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.808421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.808546 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.808851 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.808854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.808856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.809169 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:54.809378 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:54.809611 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 03:15:54.809857 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":251,"threads_started":1}
I20260924 03:15:54.811467 21194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:54.891551 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 21071
I20260924 03:15:54.906395 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 21204
Sep 24 03:15:54 dist-test-slave-vd0p krb5kdc[20831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:15:54.921306 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:54.907233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.14.227.131:37035)
0924 03:15:54.907774 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:54.907781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:54.907808 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:15:54.907867 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:54.907872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:54.908067 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:54.908229 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:54.909191 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.910158 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.911425 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:54.911985 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:54.912118 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:54.916013 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:54.916023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:54.916036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:54.916066 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:54.918688 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.919367 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.919374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.919376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.919498 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:54.919862 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:54.919867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:54.919870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:54.920184 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:54.920526 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:54.920781 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 03:15:54.921024 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260924 03:15:54.922534 21327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:55.004495 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 21204
I20260924 03:15:55.019838 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 20844
I20260924 03:15:55.020944 20906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:55.123531 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 20844
2026-09-24T03:15:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3776 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:15:55 dist-test-slave-vd0p krb5kdc[21359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:15:55 dist-test-slave-vd0p krb5kdc[21359](info): set up 2 sockets
Sep 24 03:15:55 dist-test-slave-vd0p krb5kdc[21359](info): commencing operation
krb5kdc: starting...
W20260924 03:15:56.206565 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:15:56 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:15:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.190.keytab.
Entry for principal oryx/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.190.keytab.
I20260924 03:15:56.402086 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43857
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.190.keytab
--principal=oryx/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:43857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:56.708830 21372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:56.709107 21372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:56.709189 21372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:56.716197 21372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:56.716497 21372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:56.716594 21372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:56.716673 21372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:56.716728 21372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:56.727631 21372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:43857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.190.keytab
--principal=oryx/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:56.728924 21372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:56.730239 21372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:56.742532 21377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:56.742583 21378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:56.742746 21380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:56.743194 21372 server_base.cc:1034] running on GCE node
I20260924 03:15:56.743971 21372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:56.745285 21372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:56.746479 21372 hybrid_clock.cc:648] HybridClock initialized: now 1790219756746432 us; error 58 us; skew 500 ppm
I20260924 03:15:56.746809 21372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:56 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:56.749859 21372 init.cc:377] Logged in from keytab as oryx/127.14.227.190@KRBTEST.COM (short username oryx)
I20260924 03:15:56.751186 21372 webserver.cc:460] Webserver started at http://127.14.227.190:34259/ using document root <none> and password file <none>
I20260924 03:15:56.751817 21372 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:56.751958 21372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:56.752238 21372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:56.755074 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "c892c61c6e3f48c0a38a771ea208748d"
format_stamp: "Formatted at 2026-09-24 03:15:56 on dist-test-slave-vd0p"
server_key: "84829eff5ca01a336dde5a71a1b5ec78"
server_key_iv: "901a80d754cb1463dc3bc252ccc6557d"
server_key_version: "encryptionkey@0"
I20260924 03:15:56.755815 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "c892c61c6e3f48c0a38a771ea208748d"
format_stamp: "Formatted at 2026-09-24 03:15:56 on dist-test-slave-vd0p"
server_key: "84829eff5ca01a336dde5a71a1b5ec78"
server_key_iv: "901a80d754cb1463dc3bc252ccc6557d"
server_key_version: "encryptionkey@0"
I20260924 03:15:56.760563 21372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:15:56.763693 21387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:56.764945 21372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:15:56.765071 21372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "c892c61c6e3f48c0a38a771ea208748d"
format_stamp: "Formatted at 2026-09-24 03:15:56 on dist-test-slave-vd0p"
server_key: "84829eff5ca01a336dde5a71a1b5ec78"
server_key_iv: "901a80d754cb1463dc3bc252ccc6557d"
server_key_version: "encryptionkey@0"
I20260924 03:15:56.765223 21372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:56.782408 21372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:56.786628 21372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:56.786854 21372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:56.804095 21372 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:43857
I20260924 03:15:56.804117 21438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:43857 every 8 connection(s)
I20260924 03:15:56.805428 21372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:15:56.809118 21439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:56.809415 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 21372
I20260924 03:15:56.809617 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:15:56.810357 15246 external_mini_cluster.cc:1428] Setting key aea8b4d5768a301947f4705b8b9fc652
I20260924 03:15:56.822331 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: Bootstrap starting.
Sep 24 03:15:56 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:15:56.825639 21439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:56.826776 21439 log.cc:824] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:56.829445 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: No bootstrap required, opened a new log
I20260924 03:15:56.832352 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:56.812177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.14.227.190:43857)
0924 03:15:56.812890 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:56.812901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:56.812943 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:15:56.814870 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:56.814890 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:56.815271 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:56.815581 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:56.817251 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:56.818608 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:56.820347 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:56.820972 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:56.821100 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:56.825445 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:56.825452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:56.825465 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:56.825496 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:56.828557 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:56.829150 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:56.829172 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:56.829178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:56.829334 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:56.829740 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:56.829766 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:56.829770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:56.830320 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:56.830648 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:56.831065 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 03:15:56.831440 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":330,"threads_started":1}
I20260924 03:15:56.837138 21439 raft_consensus.cc:348] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } }
I20260924 03:15:56.837519 21439 raft_consensus.cc:374] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:56.837620 21439 raft_consensus.cc:729] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c892c61c6e3f48c0a38a771ea208748d, State: Initialized, Role: FOLLOWER
I20260924 03:15:56.838265 21439 consensus_queue.cc:260] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } }
I20260924 03:15:56.838493 21439 raft_consensus.cc:388] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:15:56.838589 21439 raft_consensus.cc:482] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:15:56.838711 21439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:56.840663 21439 raft_consensus.cc:504] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } }
I20260924 03:15:56.841061 21439 leader_election.cc:302] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c892c61c6e3f48c0a38a771ea208748d; no voters:
I20260924 03:15:56.841554 21439 leader_election.cc:288] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:15:56.841678 21444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:15:56.842392 21444 raft_consensus.cc:686] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 1 LEADER]: Becoming Leader. State: Replica: c892c61c6e3f48c0a38a771ea208748d, State: Running, Role: LEADER
I20260924 03:15:56.842900 21444 consensus_queue.cc:237] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } }
I20260924 03:15:56.843539 21439 sys_catalog.cc:564] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:56.846336 21445 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } }
I20260924 03:15:56.846797 21445 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:56.846271 21446 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c892c61c6e3f48c0a38a771ea208748d. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } }
I20260924 03:15:56.847766 21446 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
W20260924 03:15:56.855605 21460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:15:56.855734 21460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:15:56.855897 21453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:15:56.857371 21453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:15:56.864648 21453 catalog_manager.cc:1347] Generated new cluster ID: fb31b95fcf71478bbfc2a060611c62e9
I20260924 03:15:56.864754 21453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:15:56.898098 21453 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:15:56.899353 21453 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:15:56.905630 21453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: Generated new TSK 0
I20260924 03:15:56.906468 21453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab.
Entry for principal oryx/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab.
I20260924 03:15:56.959648 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43857
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab
--principal=oryx/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:57.283128 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:57.283380 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:57.283551 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:57.290467 21465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:57.290576 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:57.290799 21465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:15:57.299546 21465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab
--principal=oryx/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:57.300740 21465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:57.302042 21465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:57.315048 21471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:57.315303 21470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:57.315732 21473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:57.315964 21465 server_base.cc:1034] running on GCE node
I20260924 03:15:57.316553 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:57.317343 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:57.318521 21465 hybrid_clock.cc:648] HybridClock initialized: now 1790219757318473 us; error 45 us; skew 500 ppm
I20260924 03:15:57.318770 21465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:57 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219757, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:57.321731 21465 init.cc:377] Logged in from keytab as oryx/127.14.227.129@KRBTEST.COM (short username oryx)
I20260924 03:15:57.323006 21465 webserver.cc:460] Webserver started at http://127.14.227.129:46611/ using document root <none> and password file <none>
I20260924 03:15:57.323571 21465 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:57.323673 21465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:57.323936 21465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:57.326609 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "98785d9ec789413a801ce9bdff5ced2c"
format_stamp: "Formatted at 2026-09-24 03:15:57 on dist-test-slave-vd0p"
server_key: "18f774892281be7180c1708a54f7f487"
server_key_iv: "92ab0e650ad00a294c72d8876af63008"
server_key_version: "encryptionkey@0"
I20260924 03:15:57.327296 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "98785d9ec789413a801ce9bdff5ced2c"
format_stamp: "Formatted at 2026-09-24 03:15:57 on dist-test-slave-vd0p"
server_key: "18f774892281be7180c1708a54f7f487"
server_key_iv: "92ab0e650ad00a294c72d8876af63008"
server_key_version: "encryptionkey@0"
I20260924 03:15:57.332057 21465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:15:57.335299 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:57.336496 21465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:15:57.336654 21465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "98785d9ec789413a801ce9bdff5ced2c"
format_stamp: "Formatted at 2026-09-24 03:15:57 on dist-test-slave-vd0p"
server_key: "18f774892281be7180c1708a54f7f487"
server_key_iv: "92ab0e650ad00a294c72d8876af63008"
server_key_version: "encryptionkey@0"
I20260924 03:15:57.336834 21465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:57.355098 21465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:57.359190 21465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:57.359530 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:57.360387 21465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:57.361586 21465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:57.361655 21465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:57.361817 21465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:57.361876 21465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:57.398180 21465 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:40295
I20260924 03:15:57.398187 21592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:40295 every 8 connection(s)
I20260924 03:15:57.399448 21465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:15:57.409256 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 21465
I20260924 03:15:57.409464 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:15:57.410161 15246 external_mini_cluster.cc:1428] Setting key 32dd5ea308ab945baaeb5aa07edddead
Sep 24 03:15:57 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219757, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:15:57.418485 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:57.401865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:49521 (local address 127.14.227.190:43857)
0924 03:15:57.402288 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:57.402292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:57.403229 (+ 937us) server_negotiation.cc:407] Connection header received
0924 03:15:57.404916 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:57.404920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:57.404982 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:57.405078 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:57.406512 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:57.407221 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:57.408606 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:57.409004 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:57.409147 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:57.412929 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:57.412935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:57.412947 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:57.412975 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:57.415517 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:57.416183 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:57.416190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:57.416194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:57.416308 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:57.416788 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:57.416793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:57.416796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:57.417082 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:57.417255 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:57.417912 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 03:15:57.418147 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":211,"threads_started":1}
I20260924 03:15:57.419391 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:57.402427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.129:49521)
0924 03:15:57.403075 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:57.403124 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:57.404658 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:57.405306 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:57.405313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:57.405776 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:57.406365 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:57.406382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:57.407336 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:57.407339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:57.408481 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:57.408493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:57.409235 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:57.409242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:57.409604 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:57.410388 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:57.410423 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:57.412706 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:57.415653 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:57.415662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:57.415675 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:57.416061 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:57.416414 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:57.416438 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:57.416442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:57.416675 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:57.417229 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:57.417236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:57.417554 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 03:15:57.417804 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 03:15:57.418356 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":234,"threads_started":1}
I20260924 03:15:57.420848 21593 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43857
I20260924 03:15:57.421160 21593 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:57.421985 21593 heartbeater.cc:511] Master 127.14.227.190:43857 requested a full tablet report, sending...
I20260924 03:15:57.424348 21404 ts_manager.cc:194] Registered new tserver with Master: 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129:40295)
I20260924 03:15:57.426455 21404 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:49521
WARNING: no policy specified for oryx/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab.
Entry for principal oryx/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab.
I20260924 03:15:57.453459 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43857
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab
--principal=oryx/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:57.795107 21599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:57.795367 21599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:57.795507 21599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:57.802454 21599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:57.802556 21599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:57.802753 21599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:15:57.812314 21599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab
--principal=oryx/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:57.813494 21599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:57.814810 21599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:57.828490 21604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:57.828492 21605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:57.829499 21607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:57.830013 21599 server_base.cc:1034] running on GCE node
I20260924 03:15:57.830590 21599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:57.831498 21599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:57.832738 21599 hybrid_clock.cc:648] HybridClock initialized: now 1790219757832682 us; error 41 us; skew 500 ppm
I20260924 03:15:57.833061 21599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:57 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219757, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:57.836262 21599 init.cc:377] Logged in from keytab as oryx/127.14.227.130@KRBTEST.COM (short username oryx)
I20260924 03:15:57.837649 21599 webserver.cc:460] Webserver started at http://127.14.227.130:39717/ using document root <none> and password file <none>
I20260924 03:15:57.838342 21599 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:57.838471 21599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:57.838748 21599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:57.841601 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "96be3ec41d44425a96b119bb6978e3ff"
format_stamp: "Formatted at 2026-09-24 03:15:57 on dist-test-slave-vd0p"
server_key: "cde951a968107675659a841b50579f1a"
server_key_iv: "c1bbc3054a9b15d4e118389b22290bd1"
server_key_version: "encryptionkey@0"
I20260924 03:15:57.842381 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "96be3ec41d44425a96b119bb6978e3ff"
format_stamp: "Formatted at 2026-09-24 03:15:57 on dist-test-slave-vd0p"
server_key: "cde951a968107675659a841b50579f1a"
server_key_iv: "c1bbc3054a9b15d4e118389b22290bd1"
server_key_version: "encryptionkey@0"
I20260924 03:15:57.847323 21599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 03:15:57.850612 21614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:57.851863 21599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:15:57.852082 21599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "96be3ec41d44425a96b119bb6978e3ff"
format_stamp: "Formatted at 2026-09-24 03:15:57 on dist-test-slave-vd0p"
server_key: "cde951a968107675659a841b50579f1a"
server_key_iv: "c1bbc3054a9b15d4e118389b22290bd1"
server_key_version: "encryptionkey@0"
I20260924 03:15:57.852269 21599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:57.868284 21599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:57.871973 21599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:57.872257 21599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:57.873081 21599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:57.874356 21599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:57.874428 21599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:57.874599 21599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:57.874650 21599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:57.906314 21599 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:41757
I20260924 03:15:57.906289 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:41757 every 8 connection(s)
I20260924 03:15:57.907613 21599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:15:57.911403 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 21599
I20260924 03:15:57.911573 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:15:57.912147 15246 external_mini_cluster.cc:1428] Setting key e7c37b83423a5c5f4fb0ae317a7db530
Sep 24 03:15:57 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219757, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:15:57.929718 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:57.910023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:60787 (local address 127.14.227.190:43857)
0924 03:15:57.910197 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:57.910201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:57.911443 (+ 1242us) server_negotiation.cc:407] Connection header received
0924 03:15:57.913601 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:57.913606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:57.913671 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:57.913762 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:57.915744 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:57.916501 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:57.918904 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:57.919282 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:57.919376 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:57.923636 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:57.923641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:57.923644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:57.923671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:57.926193 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:57.927180 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:57.927184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:57.927186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:57.927289 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:57.927838 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:57.927840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:57.927842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:57.928093 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:57.928213 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:57.929140 (+ 927us) server_negotiation.cc:299] Negotiation successful
0924 03:15:57.929324 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:15:57.931223 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:57.910511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.130:60787)
0924 03:15:57.911242 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:57.911298 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:57.913315 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:57.914081 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:57.914089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:57.914783 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:57.915556 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:57.915594 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:57.916664 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:57.916672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:57.918644 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:57.918660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:57.919514 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:57.919523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:57.919861 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:57.920697 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:57.920737 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:57.923397 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:57.926326 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:57.926338 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:57.926355 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:57.927058 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:57.927401 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:57.927409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:57.927416 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:57.927705 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:57.928246 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:57.928275 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:57.928740 (+ 465us) client_negotiation.cc:769] Sending connection context
0924 03:15:57.929067 (+ 327us) client_negotiation.cc:240] Negotiation successful
0924 03:15:57.929645 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":248,"threads_started":1}
I20260924 03:15:57.932633 21727 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43857
I20260924 03:15:57.932993 21727 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:57.933919 21727 heartbeater.cc:511] Master 127.14.227.190:43857 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.14.227.131@KRBTEST.COM; defaulting to no policy
I20260924 03:15:57.935505 21404 ts_manager.cc:194] Registered new tserver with Master: 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757)
I20260924 03:15:57.936342 21404 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:60787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab.
Entry for principal oryx/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab.
I20260924 03:15:57.961649 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43857
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab
--principal=oryx/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:58.264334 21732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:58.264576 21732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:58.264768 21732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:58.271721 21732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:58.271812 21732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:58.271989 21732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:15:58.280954 21732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab
--principal=oryx/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:58.282193 21732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:58.283463 21732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:58.296940 21738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:58.296983 21740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:58.297025 21737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:58.303256 21732 server_base.cc:1034] running on GCE node
I20260924 03:15:58.303963 21732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:58.304877 21732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:58.306106 21732 hybrid_clock.cc:648] HybridClock initialized: now 1790219758306053 us; error 45 us; skew 500 ppm
I20260924 03:15:58.306440 21732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:58 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219758, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:58.309454 21732 init.cc:377] Logged in from keytab as oryx/127.14.227.131@KRBTEST.COM (short username oryx)
I20260924 03:15:58.311026 21732 webserver.cc:460] Webserver started at http://127.14.227.131:38683/ using document root <none> and password file <none>
I20260924 03:15:58.311782 21732 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:58.311928 21732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:58.312227 21732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:58.315095 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "bb775656b7d9489fa30ee0eeab6484bb"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "9dc4bdc11e92cf90ce7dfcf58b334731"
server_key_iv: "0ed754dc1562126a19ff543d84f37db5"
server_key_version: "encryptionkey@0"
I20260924 03:15:58.316030 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "bb775656b7d9489fa30ee0eeab6484bb"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "9dc4bdc11e92cf90ce7dfcf58b334731"
server_key_iv: "0ed754dc1562126a19ff543d84f37db5"
server_key_version: "encryptionkey@0"
I20260924 03:15:58.322013 21732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:15:58.325492 21747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:58.326823 21732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:15:58.326987 21732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "bb775656b7d9489fa30ee0eeab6484bb"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "9dc4bdc11e92cf90ce7dfcf58b334731"
server_key_iv: "0ed754dc1562126a19ff543d84f37db5"
server_key_version: "encryptionkey@0"
I20260924 03:15:58.327175 21732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:58.359606 21732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:58.363212 21732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:58.363492 21732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:58.364413 21732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:15:58.365646 21732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:15:58.365744 21732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:58.365881 21732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:15:58.365973 21732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:58.397625 21732 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:38741
I20260924 03:15:58.397644 21859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:38741 every 8 connection(s)
I20260924 03:15:58.399150 21732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:15:58.409073 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 21732
I20260924 03:15:58.409264 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:15:58.409794 15246 external_mini_cluster.cc:1428] Setting key b7ee97eb34b8e5bae457d6dfa1196d1b
Sep 24 03:15:58 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219758, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:15:58.417918 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.401694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:39579 (local address 127.14.227.190:43857)
0924 03:15:58.401863 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:58.401867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:58.403144 (+ 1277us) server_negotiation.cc:407] Connection header received
0924 03:15:58.404882 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:58.404887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:58.405001 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:58.405111 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:58.406537 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.407262 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.408815 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.409275 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.409406 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.413019 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:58.413023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:58.413027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:58.413055 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:58.415421 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.416041 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.416045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.416047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.416167 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.416516 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.416518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.416520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.416863 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:58.416972 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:58.417536 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 03:15:58.417683 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:15:58.418817 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.402286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.131:39579)
0924 03:15:58.402983 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:58.403021 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:58.404626 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:58.405358 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:58.405365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:58.405800 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:58.406375 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.406391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.407408 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.407418 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.408662 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.408693 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.409484 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.409492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.409749 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.410412 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:58.410432 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:58.412826 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:58.415531 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.415536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.415546 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.415938 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.416252 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.416255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.416257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.416416 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.416967 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:58.416972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:58.417236 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 03:15:58.417438 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 03:15:58.417774 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":305,"threads_started":1}
I20260924 03:15:58.420020 21860 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43857
I20260924 03:15:58.420254 21860 heartbeater.cc:464] Registering TS with master...
I20260924 03:15:58.420878 21860 heartbeater.cc:511] Master 127.14.227.190:43857 requested a full tablet report, sending...
I20260924 03:15:58.422183 21404 ts_manager.cc:194] Registered new tserver with Master: bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131:38741)
I20260924 03:15:58.422942 21404 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:39579
I20260924 03:15:58.425031 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:15:58.429998 21593 heartbeater.cc:503] Master 127.14.227.190:43857 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.14.227.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.189.keytab.
Entry for principal oryx/127.14.227.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.189.keytab.
I20260924 03:15:58.469810 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.189:44787
--webserver_interface=127.14.227.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.189.keytab
--principal=oryx/127.14.227.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:43857,127.14.227.189:44787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:15:58.783180 21865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:58.783444 21865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:58.783648 21865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:15:58.790638 21865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:58.790769 21865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:58.790871 21865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:58.790973 21865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:58.791056 21865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:58.800540 21865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:43857,127.14.227.189:44787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.189.keytab
--principal=oryx/127.14.227.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.189:44787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.227.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
W20260924 03:15:58.800858 21865 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.227.190:43857,127.14.227.189:44787'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 03:15:58.801821 21865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:58.803151 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:58.814577 21873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:58.814617 21871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:58.814901 21870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:58.815853 21865 server_base.cc:1034] running on GCE node
I20260924 03:15:58.816552 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:58.817808 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:58.819072 21865 hybrid_clock.cc:648] HybridClock initialized: now 1790219758819005 us; error 69 us; skew 500 ppm
I20260924 03:15:58.819440 21865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:58 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219758, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:58.822937 21865 init.cc:377] Logged in from keytab as oryx/127.14.227.189@KRBTEST.COM (short username oryx)
I20260924 03:15:58.827006 21865 webserver.cc:460] Webserver started at http://127.14.227.189:36719/ using document root <none> and password file <none>
I20260924 03:15:58.827646 21865 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:58.827769 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:58.828099 21865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:58.831060 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data/instance:
uuid: "6fff4654be144b74bb03b1adf4dfca63"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "e6d618e1fdc1aa0b416af4a960dda0b7"
server_key_iv: "49a70c8d7220ac493f7cbc0386c99458"
server_key_version: "encryptionkey@0"
I20260924 03:15:58.831910 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal/instance:
uuid: "6fff4654be144b74bb03b1adf4dfca63"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "e6d618e1fdc1aa0b416af4a960dda0b7"
server_key_iv: "49a70c8d7220ac493f7cbc0386c99458"
server_key_version: "encryptionkey@0"
I20260924 03:15:58.836777 21865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 03:15:58.840013 21880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:58.841282 21865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:15:58.841450 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
uuid: "6fff4654be144b74bb03b1adf4dfca63"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "e6d618e1fdc1aa0b416af4a960dda0b7"
server_key_iv: "49a70c8d7220ac493f7cbc0386c99458"
server_key_version: "encryptionkey@0"
I20260924 03:15:58.841681 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:58.866257 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:58.870221 21865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:58.870469 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:58.887961 21865 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.189:44787
I20260924 03:15:58.888002 21931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.189:44787 every 8 connection(s)
I20260924 03:15:58.889249 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
I20260924 03:15:58.892962 21932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:58.897789 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 21865
I20260924 03:15:58.900125 21932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:15:58 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219758, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.189@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
Sep 24 03:15:58 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.189@KRBTEST.COM
I20260924 03:15:58.918569 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.902758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.14.227.190:43857)
0924 03:15:58.902938 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:58.902944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:58.903617 (+ 673us) server_negotiation.cc:407] Connection header received
0924 03:15:58.903895 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:58.903901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:58.903980 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:58.904093 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:58.905234 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.906079 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.907931 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.908580 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.908709 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.912271 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:58.912276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:58.912279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:58.912308 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:58.915443 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.916209 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.916214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.916217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.916371 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.916754 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.916759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.916765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.917143 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:58.917241 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:58.918043 (+ 802us) server_negotiation.cc:299] Negotiation successful
0924 03:15:58.918329 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:15:58.919790 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.902795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:58436)
0924 03:15:58.903480 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:58.903537 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:58.903738 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:58.904161 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:58.904165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:58.904703 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:58.905059 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.905081 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.906204 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.906209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.907790 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.907804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.908790 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.908797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.909093 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.909921 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:58.909973 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:58.912098 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:58.915572 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.915581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.915588 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.916082 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.916478 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.916480 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.916482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.916650 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.917257 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:58.917264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:58.917654 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 03:15:58.917914 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 03:15:58.918434 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":54,"thread_start_us":325,"threads_started":1}
I20260924 03:15:58.920251 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.900074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.14.227.189:44787)
0924 03:15:58.900853 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:58.900864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:58.900901 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:15:58.902742 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:58.902761 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:58.903282 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:58.903644 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:58.905108 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.906773 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.908329 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.909032 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.909141 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.913645 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:58.913658 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:58.913674 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:58.913707 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:58.916944 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.917636 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.917641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.917643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.917755 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.918240 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.918245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.918254 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.918638 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:58.918916 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:58.919539 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 03:15:58.919745 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":544,"thread_start_us":366,"threads_started":1}
I20260924 03:15:58.921747 21932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:15:58 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219758, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.189@KRBTEST.COM for oryx/127.14.227.189@KRBTEST.COM
I20260924 03:15:58.932786 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.922366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.0.0.1:47866)
0924 03:15:58.922540 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:58.922559 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:58.922658 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:58.922915 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:58.922919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:58.923227 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:58.923333 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.923339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.924269 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.924274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.925126 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.925136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.925789 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.925795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.926045 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.926493 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:58.926507 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:58.928222 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:58.930907 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.930910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.930912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.931273 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.931647 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.931654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.931656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.931772 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.932168 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:58.932171 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:58.932283 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 03:15:58.932380 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:15:58.932562 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 03:15:58.932931 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.922400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.14.227.189:44787)
0924 03:15:58.922545 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:58.922551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:58.922603 (+ 52us) server_negotiation.cc:407] Connection header received
0924 03:15:58.922763 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:58.922767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:58.922816 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:58.922902 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:58.923449 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.924136 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.925257 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.925647 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.925784 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.928373 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:58.928377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:58.928380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:58.928404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:58.930809 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.931376 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.931382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.931385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.931536 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.931854 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.931856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.931858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.932081 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:58.932174 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:58.932455 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 03:15:58.932667 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":72}
I20260924 03:15:58.937043 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Bootstrap starting.
I20260924 03:15:58.939074 21727 heartbeater.cc:503] Master 127.14.227.190:43857 was elected leader, sending a full tablet report...
I20260924 03:15:58.940181 21932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:58.941293 21932 log.cc:824] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:58.943790 21932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: No bootstrap required, opened a new log
I20260924 03:15:58.950837 21932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } }
I20260924 03:15:58.951114 21932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:58.951203 21932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fff4654be144b74bb03b1adf4dfca63, State: Initialized, Role: FOLLOWER
I20260924 03:15:58.951668 21932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } }
I20260924 03:15:58.952339 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } } }
I20260924 03:15:58.952687 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:58.953249 21932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:15:58.961860 21949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:15:58.962011 21949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:15:58.963712 21865 master_runner.cc:428] Detected that this master 6fff4654be144b74bb03b1adf4dfca63 is joining an existing cluster
I20260924 03:15:58.963809 21865 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.227.189:44787
I20260924 03:15:58.986655 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.967914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.14.227.190:43857)
0924 03:15:58.968148 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:58.968156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:58.970536 (+ 2380us) server_negotiation.cc:407] Connection header received
0924 03:15:58.970697 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:58.970702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:58.970768 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:58.970876 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:58.971886 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.972591 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.974942 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.975648 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.975762 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.979845 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:58.979872 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:58.979880 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:58.979925 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:58.983676 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.984365 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.984370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.984374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.984506 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.985072 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.985077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.985081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.985356 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:58.985501 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:58.986036 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 03:15:58.986268 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:15:58.987183 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.967472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:58446)
0924 03:15:58.970439 (+ 2967us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:58.970453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:58.970582 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:58.970901 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:58.970905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:58.971445 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:58.971747 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.971755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.972736 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.972744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.974765 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.974785 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.976068 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.976077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.976346 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.978362 (+ 2016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:58.978388 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:58.979627 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:58.983799 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.983806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.983810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.984243 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.984813 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.984817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.984819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.984949 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.985468 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:58.985472 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:58.985694 (+ 222us) client_negotiation.cc:769] Sending connection context
0924 03:15:58.985874 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 03:15:58.986684 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2842,"mutex_wait_us":28,"thread_start_us":2720,"threads_started":1}
I20260924 03:15:58.989604 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.967777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.0.0.1:47882)
0924 03:15:58.974380 (+ 6603us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:58.974396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:58.974726 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:58.975145 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:58.975150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:58.975778 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:58.976212 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.976225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.977729 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.977734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.979365 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.979379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.980698 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.980704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.980865 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.981435 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:58.981458 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:58.982505 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:58.986607 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.986612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.986616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.987205 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.987873 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:58.987877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:58.987880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:58.988057 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:58.988613 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:58.988617 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:58.988758 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 03:15:58.988898 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 03:15:58.989158 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6478,"mutex_wait_us":64,"thread_start_us":6700,"threads_started":1}
I20260924 03:15:58.990435 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.967501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.14.227.189:44787)
0924 03:15:58.967689 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:58.967697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:58.974473 (+ 6776us) server_negotiation.cc:407] Connection header received
0924 03:15:58.974855 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:58.974860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:58.974920 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:58.974981 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:58.976350 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.977527 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.979539 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.980313 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.980409 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:58.982663 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:58.982674 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:58.982679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:58.982710 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:58.986472 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.987593 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.987599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.987602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.987740 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:58.988176 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:58.988180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:58.988183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:58.988497 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:58.988659 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:58.989880 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0924 03:15:58.990096 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":52}
Sep 24 03:15:58 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.190@KRBTEST.COM for oryx/127.14.227.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.227.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.188@KRBTEST.COM" created.
I20260924 03:15:59.010313 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.994165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.14.227.189:44787)
0924 03:15:58.994328 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:58.994333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:58.995085 (+ 752us) server_negotiation.cc:407] Connection header received
0924 03:15:58.995403 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:58.995408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:58.995533 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:58.995635 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:58.996665 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.997383 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.998815 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.999289 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.999437 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.002705 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.002713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.002717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.002745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.006200 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.007104 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.007110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.007113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.007270 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.007718 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.007723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.007727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.008040 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.008195 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.009696 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.009920 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:15:59.011566 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:58.994318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.0.0.1:47894)
0924 03:15:58.994969 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:58.995003 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:58.995203 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:58.995678 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:58.995685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:58.996320 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:58.996519 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.996528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.997583 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.997589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.998661 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:58.998673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:58.999447 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:58.999453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:58.999657 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.000271 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.000287 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.002535 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.006335 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.006349 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.006360 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.006962 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.007390 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.007399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.007405 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.007585 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.008185 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.009104 (+ 919us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.009327 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.009552 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.009922 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":314,"threads_started":1}
I20260924 03:15:59.013478 21404 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.227.189:44787: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63"
member_type: NON_VOTER
last_known_addr {
host: "127.14.227.189"
port: 44787
}
attrs {
promote: true
}
}
dest_uuid: "c892c61c6e3f48c0a38a771ea208748d"
cas_config_opid_index: -1
I20260924 03:15:59.014493 21404 consensus_queue.cc:237] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } }
I20260924 03:15:59.027786 21961 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6fff4654be144b74bb03b1adf4dfca63 (127.14.227.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } }
I20260924 03:15:59.029330 21907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:15:59.032022 21961 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.227.189:44787
I20260924 03:15:59.032682 21865 master.cc:537] Master@127.14.227.189:44787 shutting down...
I20260924 03:15:59.033098 21907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Refusing update from remote peer c892c61c6e3f48c0a38a771ea208748d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 03:15:59.032544 21961 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } }
I20260924 03:15:59.033836 21961 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:59.034266 21962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.188.keytab.
Entry for principal oryx/127.14.227.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.188.keytab.
W20260924 03:15:59.037050 21389 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 03:15:59.038235 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.188:42013
--webserver_interface=127.14.227.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.188.keytab
--principal=oryx/127.14.227.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:43857,127.14.227.189:44787,127.14.227.188:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:15:59.048723 21865 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 03:15:59.049021 21389 consensus_peers.cc:487] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d -> Peer 6fff4654be144b74bb03b1adf4dfca63 (127.14.227.189:44787): Couldn't send request to peer 6fff4654be144b74bb03b1adf4dfca63. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:15:59.049237 21865 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:59.049471 21865 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: stopping tablet replica
I20260924 03:15:59.059621 21865 master.cc:552] Master@127.14.227.189:44787 shutdown complete.
I20260924 03:15:59.059768 21865 master_runner.cc:305] Clearing existing system tablet
I20260924 03:15:59.060189 21865 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:59.060289 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:59.063292 21865 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 03:15:59.074285 21967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:59.074653 21865 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:59.074929 21865 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.007s sys 0.004s
I20260924 03:15:59.075018 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
uuid: "6fff4654be144b74bb03b1adf4dfca63"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "e6d618e1fdc1aa0b416af4a960dda0b7"
server_key_iv: "49a70c8d7220ac493f7cbc0386c99458"
server_key_version: "encryptionkey@0"
I20260924 03:15:59.079864 21865 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:59.086463 21865 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:15:59.086647 21865 log.cc:1196] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 03:15:59.087091 21865 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Deleting consensus metadata
I20260924 03:15:59.087348 21865 master_runner.cc:315] Copying system tablet from 127.14.227.190:43857
I20260924 03:15:59.089655 21865 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: tablet copy: Beginning tablet copy session from remote peer at address 127.14.227.190:43857
I20260924 03:15:59.105391 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.090793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.14.227.190:43857)
0924 03:15:59.090949 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.090955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.091092 (+ 137us) server_negotiation.cc:407] Connection header received
0924 03:15:59.091287 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.091292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.091382 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.091487 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.092514 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.093625 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.095522 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.096154 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.096281 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.098595 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.098602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.098607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.098645 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.102253 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.103154 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.103160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.103164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.103301 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.103788 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.103793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.103796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.104148 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.104295 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.104802 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.105043 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:15:59.105526 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.090403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:58460)
0924 03:15:59.090970 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.090995 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.091145 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.091565 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.091570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.092074 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.092340 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.092355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.093801 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.093806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.095355 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.095369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.096329 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.096338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.096568 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.097312 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.097335 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.098379 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.102396 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.102402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.102406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.103023 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.103433 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.103440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.103445 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.103661 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.104290 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.104298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.104504 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.104702 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.105108 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":436,"threads_started":1}
I20260924 03:15:59.106707 21424 tablet_copy_service.cc:140] P c892c61c6e3f48c0a38a771ea208748d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6fff4654be144b74bb03b1adf4dfca63 ({username='oryx', principal='oryx/127.14.227.189@KRBTEST.COM'} at 127.0.0.1:58460)
I20260924 03:15:59.107012 21424 tablet_copy_service.cc:161] P c892c61c6e3f48c0a38a771ea208748d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6fff4654be144b74bb03b1adf4dfca63 at {username='oryx', principal='oryx/127.14.227.189@KRBTEST.COM'} at 127.0.0.1:58460: session id = 6fff4654be144b74bb03b1adf4dfca63-00000000000000000000000000000000
I20260924 03:15:59.111487 21424 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:15:59.113595 21865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:15:59.120100 21865 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: tablet copy: Starting download of 0 data blocks...
I20260924 03:15:59.120419 21865 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: tablet copy: Starting download of 1 WAL segments...
I20260924 03:15:59.123165 21865 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:15:59.127563 21424 tablet_copy_service.cc:342] P c892c61c6e3f48c0a38a771ea208748d: Request end of tablet copy session 6fff4654be144b74bb03b1adf4dfca63-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.227.189@KRBTEST.COM'} at 127.0.0.1:58460
I20260924 03:15:59.127745 21424 tablet_copy_service.cc:433] P c892c61c6e3f48c0a38a771ea208748d: ending tablet copy session 6fff4654be144b74bb03b1adf4dfca63-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6fff4654be144b74bb03b1adf4dfca63
I20260924 03:15:59.138692 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:59.141412 21974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:59.142158 21975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:59.154038 21865 server_base.cc:1034] running on GCE node
W20260924 03:15:59.154152 21977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:59.154587 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:59.155290 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:59.156493 21865 hybrid_clock.cc:648] HybridClock initialized: now 1790219759156421 us; error 43 us; skew 500 ppm
I20260924 03:15:59.156751 21865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:59.159243 21865 init.cc:377] Logged in from keytab as oryx/127.14.227.189@KRBTEST.COM (short username oryx)
I20260924 03:15:59.162906 21865 webserver.cc:460] Webserver started at http://127.14.227.189:43617/ using document root <none> and password file <none>
I20260924 03:15:59.163264 21865 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:59.163345 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:59.166460 21865 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 03:15:59.168337 21984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:59.168900 21865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260924 03:15:59.169032 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
uuid: "6fff4654be144b74bb03b1adf4dfca63"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "e6d618e1fdc1aa0b416af4a960dda0b7"
server_key_iv: "49a70c8d7220ac493f7cbc0386c99458"
server_key_version: "encryptionkey@0"
I20260924 03:15:59.169234 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:59.184065 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:59.187794 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:59.207970 21865 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.189:44787
I20260924 03:15:59.208015 22035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.189:44787 every 8 connection(s)
I20260924 03:15:59.208901 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
I20260924 03:15:59.214429 22036 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:15:59.218464 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Bootstrap starting.
I20260924 03:15:59.235368 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:15:59.235852 22036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: Bootstrap complete.
I20260924 03:15:59.236805 22036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } }
I20260924 03:15:59.237066 22036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6fff4654be144b74bb03b1adf4dfca63, State: Initialized, Role: LEARNER
I20260924 03:15:59.237283 22036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } }
I20260924 03:15:59.238075 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } }
I20260924 03:15:59.238435 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:15:59.238477 22036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:59.247143 22050 catalog_manager.cc:1259] Loaded cluster ID: fb31b95fcf71478bbfc2a060611c62e9
I20260924 03:15:59.247242 22050 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: loading cluster ID for follower catalog manager: success
I20260924 03:15:59.250459 22050 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: acquiring CA information for follower catalog manager: success
I20260924 03:15:59.252683 22050 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 03:15:59.418293 21965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:15:59.418589 21965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:15:59.418730 21965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 03:15:59.425706 21860 heartbeater.cc:503] Master 127.14.227.190:43857 was elected leader, sending a full tablet report...
W20260924 03:15:59.426263 21965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:15:59.426393 21965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:15:59.426481 21965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:15:59.426586 21965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:15:59.426664 21965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:15:59.436632 21965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:43857,127.14.227.189:44787,127.14.227.188:42013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.188.keytab
--principal=oryx/127.14.227.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.188:42013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.227.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:15:59.437862 21965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:15:59.439169 21965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:59.452976 22055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:59.453330 22058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:59.454537 22056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:59.454614 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.445847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.0.0.1:47902)
0924 03:15:59.446115 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.446131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.446216 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.448037 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.448044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.448078 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:15:59.448251 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.448276 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.450039 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.450043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.451927 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.451957 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.453714 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.453723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.453930 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.454022 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.454124 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.454290 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 03:15:59.454715 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.447045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.14.227.189:44787)
0924 03:15:59.447660 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.447666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.447689 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:15:59.447758 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.447764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.447848 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.447967 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:15:59.448394 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.449869 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.452131 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.453583 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.453757 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.453935 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.454179 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.454363 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":334,"threads_started":1}
I20260924 03:15:59.455993 22011 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 03:15:59.456127 21965 server_base.cc:1034] running on GCE node
I20260924 03:15:59.456593 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c892c61c6e3f48c0a38a771ea208748d. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } }
I20260924 03:15:59.456852 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:15:59.457103 21965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:59.458477 21965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:59.459662 21965 hybrid_clock.cc:648] HybridClock initialized: now 1790219759459615 us; error 43 us; skew 500 ppm
I20260924 03:15:59.459985 21965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:59.462903 21965 init.cc:377] Logged in from keytab as oryx/127.14.227.188@KRBTEST.COM (short username oryx)
I20260924 03:15:59.464262 21965 webserver.cc:460] Webserver started at http://127.14.227.188:42971/ using document root <none> and password file <none>
I20260924 03:15:59.464933 21965 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:59.465073 21965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:59.465355 21965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:15:59.468164 21965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data/instance:
uuid: "163c833a843448b8af4da472b811e5b5"
format_stamp: "Formatted at 2026-09-24 03:15:59 on dist-test-slave-vd0p"
server_key: "9793cbd6f4a0730854434c6b68d9237c"
server_key_iv: "60b2267534a4ae497365b462caa92e72"
server_key_version: "encryptionkey@0"
I20260924 03:15:59.468889 21965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal/instance:
uuid: "163c833a843448b8af4da472b811e5b5"
format_stamp: "Formatted at 2026-09-24 03:15:59 on dist-test-slave-vd0p"
server_key: "9793cbd6f4a0730854434c6b68d9237c"
server_key_iv: "60b2267534a4ae497365b462caa92e72"
server_key_version: "encryptionkey@0"
I20260924 03:15:59.473760 21965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 03:15:59.477401 22066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:59.478752 21965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:15:59.478919 21965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
uuid: "163c833a843448b8af4da472b811e5b5"
format_stamp: "Formatted at 2026-09-24 03:15:59 on dist-test-slave-vd0p"
server_key: "9793cbd6f4a0730854434c6b68d9237c"
server_key_iv: "60b2267534a4ae497365b462caa92e72"
server_key_version: "encryptionkey@0"
I20260924 03:15:59.479102 21965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:59.494860 21965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:59.498421 21965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:15:59.498687 21965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:59.516906 21965 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.188:42013
I20260924 03:15:59.516937 22117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.188:42013 every 8 connection(s)
I20260924 03:15:59.518229 21965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
I20260924 03:15:59.521649 22118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:15:59.527504 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 21965
I20260924 03:15:59.527685 22118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.188@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.188@KRBTEST.COM
I20260924 03:15:59.549191 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.530866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.14.227.190:43857)
0924 03:15:59.531067 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.531073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.531698 (+ 625us) server_negotiation.cc:407] Connection header received
0924 03:15:59.532996 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.533003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.533075 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.533180 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.534785 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.535928 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.537896 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.538578 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.538714 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.543006 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.543015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.543021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.543063 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.545922 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.546904 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.546908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.546910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.547042 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.547534 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.547537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.547539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.547771 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.547877 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.548687 (+ 810us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.548869 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:15:59.550288 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.530660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:58476)
0924 03:15:59.531539 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.531580 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.532756 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.533251 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.533257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.534125 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.534623 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.534645 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.536075 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.536079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.537759 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.537773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.538830 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.538836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.538984 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.539751 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.539784 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.542829 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.546087 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.546102 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.546112 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.546789 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.547186 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.547190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.547196 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.547422 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.547894 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.547904 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.548289 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.548604 (+ 315us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.549011 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":42,"thread_start_us":471,"threads_started":1}
I20260924 03:15:59.550676 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.529283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.14.227.188:42013)
0924 03:15:59.530419 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.530440 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.530884 (+ 444us) server_negotiation.cc:407] Connection header received
0924 03:15:59.532748 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.532767 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.533033 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.533218 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.534916 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.536387 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.538149 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.538917 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.539086 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.544798 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.544810 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.544825 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.544867 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.548049 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.548626 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.548634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.548637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.548762 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.549140 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.549144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.549147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.549496 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.549788 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.550038 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.550248 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":931,"spinlock_wait_cycles":12032,"thread_start_us":765,"threads_started":1}
I20260924 03:15:59.551841 22118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.188@KRBTEST.COM for oryx/127.14.227.189@KRBTEST.COM
I20260924 03:15:59.563160 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.552614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.0.0.1:47910)
0924 03:15:59.552770 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.552787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.552909 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.553267 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.553270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.553551 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.553664 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.553671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.554729 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.554733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.555662 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.555671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.556258 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.556280 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.556400 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.557007 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.557022 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.558891 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.561325 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.561328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.561330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.561635 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.561988 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.561994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.561999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.562112 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.562529 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.562532 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.562628 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.562722 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.562889 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:15:59.563185 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.552628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.14.227.189:44787)
0924 03:15:59.552805 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.552813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.552832 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:15:59.553056 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.553063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.553139 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.553217 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.553788 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.554616 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.555769 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.556154 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.556238 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.559060 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.559064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.559068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.559096 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.561227 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.561739 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.561743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.561745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.561852 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.562198 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.562201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.562202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.562441 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.562543 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.562804 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.562960 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:15:59.564069 22118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.188@KRBTEST.COM for oryx/127.14.227.188@KRBTEST.COM
I20260924 03:15:59.575186 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.564646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:42013 (local address 127.0.0.1:35218)
0924 03:15:59.564806 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.564822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.564926 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.565223 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.565228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.565563 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.565670 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.565677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.566639 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.566646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.567434 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.567444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.568085 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.568090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.568310 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.568808 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.568823 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.570815 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.573301 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.573304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.573306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.573643 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.573921 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.573923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.573926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.574112 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.574524 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.574527 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.574625 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.574733 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.574885 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 03:15:59.575217 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.564701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.14.227.188:42013)
0924 03:15:59.564821 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.564827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.564878 (+ 51us) server_negotiation.cc:407] Connection header received
0924 03:15:59.565036 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.565041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.565098 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.565188 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.565789 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.566527 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.567569 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.567960 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.568080 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.570979 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.570983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.570986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.571012 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.573198 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.573734 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.573737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.573739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.573835 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.574201 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.574204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.574206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.574433 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.574531 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.574811 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.574982 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":45}
I20260924 03:15:59.579496 22118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Bootstrap starting.
I20260924 03:15:59.582813 22118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Neither blocks nor log segments found. Creating new log.
I20260924 03:15:59.583850 22118 log.cc:824] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:15:59.586226 22118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: No bootstrap required, opened a new log
I20260924 03:15:59.592697 22118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } }
I20260924 03:15:59.592958 22118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:15:59.593034 22118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163c833a843448b8af4da472b811e5b5, State: Initialized, Role: FOLLOWER
I20260924 03:15:59.593544 22118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } }
I20260924 03:15:59.594365 22124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } } }
I20260924 03:15:59.594698 22124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:59.595676 22118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:15:59.604732 22135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:15:59.604848 22135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:15:59.604873 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 21465
I20260924 03:15:59.608405 21965 master_runner.cc:428] Detected that this master 163c833a843448b8af4da472b811e5b5 is joining an existing cluster
I20260924 03:15:59.608521 21965 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.227.188:42013
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.129@KRBTEST.COM
I20260924 03:15:59.626390 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.605862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.14.227.129:40295)
0924 03:15:59.606442 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.606448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.606469 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:15:59.606520 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.606526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.606688 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.606855 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.608460 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.610170 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.611954 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.612588 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.612722 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.618315 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.618327 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.618342 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.618380 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.622278 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.623122 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.623130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.623133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.623279 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.623780 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.623785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.623789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.624208 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.624515 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.625692 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.625978 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":297,"threads_started":1}
I20260924 03:15:59.627686 21588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:15:59.630138 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.613869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.14.227.190:43857)
0924 03:15:59.614113 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.614119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.614138 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:15:59.614192 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.614197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.614271 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.614372 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.615859 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.616961 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.619046 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.619690 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.619781 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.622770 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.622778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.622784 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.622822 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.626653 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.627381 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.627387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.627393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.627564 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.628107 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.628115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.628120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.628521 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.628652 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.629372 (+ 720us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.629646 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:15:59.631628 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.612931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:58486)
0924 03:15:59.613585 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.613603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.613722 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.614567 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.614574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.615281 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.615688 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.615702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.617144 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.617150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.618867 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.618882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.620038 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.620044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.620273 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.621022 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.621049 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.622533 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.626809 (+ 4276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.626813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.626816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.627249 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.627717 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.627725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.627729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.627935 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.628757 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.628764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.629018 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.630403 (+ 1385us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.630856 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":1010,"threads_started":1}
I20260924 03:15:59.647401 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.625180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.14.227.188:42013)
0924 03:15:59.626079 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.626084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.630046 (+ 3962us) server_negotiation.cc:407] Connection header received
0924 03:15:59.630623 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.630627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.630684 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.630737 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.632664 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.633719 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.635663 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.636359 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.636446 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.640225 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.640231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.640237 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.640268 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.643902 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.645133 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.645139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.645142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.645285 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.645687 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.645692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.645695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.646090 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.646216 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.646774 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.646973 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798}
I20260924 03:15:59.648748 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.625339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:42013 (local address 127.0.0.1:35220)
0924 03:15:59.629971 (+ 4632us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.629987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.630509 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.630921 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.630925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.631618 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.632517 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.632528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.633974 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.633979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.635536 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.635548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.636718 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.636725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.636907 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.638886 (+ 1979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.638910 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.640069 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.644073 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.644078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.644082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.645006 (+ 924us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.645428 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.645432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.645436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.645581 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.646280 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.646285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.646497 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.648116 (+ 1619us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.648365 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4490,"thread_start_us":4708,"threads_started":1}
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.190@KRBTEST.COM for oryx/127.14.227.188@KRBTEST.COM
I20260924 03:15:59.666491 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.641113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:42013 (local address 127.0.0.1:35228)
0924 03:15:59.641492 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.641513 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.641629 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.648026 (+ 6397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.648034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.648615 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.648840 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.648854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.650554 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.650562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.652289 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.652307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.654582 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.654591 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.654918 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.655492 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.655514 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.658361 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.663624 (+ 5263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.663631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.663636 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.664198 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.664675 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.664682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.664686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.664872 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.665571 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.665577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.665721 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.665840 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.666131 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260924 03:15:59.667528 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.641844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.14.227.188:42013)
0924 03:15:59.647683 (+ 5839us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.647689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.647705 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:15:59.647786 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.647791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.647881 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.647975 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.649032 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.650349 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.652668 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.654254 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.655229 (+ 975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.658582 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.658588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.658594 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.658634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.663405 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.664371 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.664378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.664382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.664534 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.665024 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.665030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.665035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.665422 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.666817 (+ 1395us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.666956 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.667157 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5761,"thread_start_us":20056,"threads_started":1}
I20260924 03:15:59.668798 21404 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.227.188:42013: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "163c833a843448b8af4da472b811e5b5"
member_type: NON_VOTER
last_known_addr {
host: "127.14.227.188"
port: 42013
}
attrs {
promote: true
}
}
dest_uuid: "c892c61c6e3f48c0a38a771ea208748d"
cas_config_opid_index: 5
I20260924 03:15:59.669855 21404 consensus_queue.cc:237] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } }
I20260924 03:15:59.670861 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.613398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.14.227.189:44787)
0924 03:15:59.613560 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.613565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.624732 (+ 11167us) server_negotiation.cc:407] Connection header received
0924 03:15:59.625962 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.625969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.626052 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.626122 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.637291 (+ 11169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.638707 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.657116 (+ 18409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.657787 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.657888 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.663628 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.663637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.663644 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.663687 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.667079 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.668592 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.668598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.668601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.668766 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.669181 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.669186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.669190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.669528 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.669640 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.670275 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.670489 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:15:59.671862 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.613487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.0.0.1:47922)
0924 03:15:59.624628 (+ 11141us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.624648 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.625679 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.626348 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.626352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.636593 (+ 10241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.637061 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.637080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.639026 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.639034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.652721 (+ 13687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.652735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.658235 (+ 5500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.658241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.658416 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.662130 (+ 3714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.662151 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.663389 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.667235 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.667240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.667243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.667734 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.668903 (+ 1169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.668907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.668911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.669059 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.669799 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.669805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.670010 (+ 205us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.671108 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.671406 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11012,"thread_start_us":11262,"threads_started":1}
I20260924 03:15:59.672355 21961 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 163c833a843448b8af4da472b811e5b5 (127.14.227.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.674726 22011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 LEARNER]: Refusing update from remote peer c892c61c6e3f48c0a38a771ea208748d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 03:15:59.675815 21961 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.227.188:42013
I20260924 03:15:59.676337 21965 master.cc:537] Master@127.14.227.188:42013 shutting down...
I20260924 03:15:59.680229 22148 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 03:15:59.686220 22147 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.686676 22147 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
W20260924 03:15:59.688987 21389 consensus_peers.cc:487] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d -> Peer 163c833a843448b8af4da472b811e5b5 (127.14.227.188:42013): Couldn't send request to peer 163c833a843448b8af4da472b811e5b5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:15:59.689980 22011 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 163c833a843448b8af4da472b811e5b5 (127.14.227.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.693122 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: NON_VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: true } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.693400 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:15:59.693717 22147 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: attempting to promote NON_VOTER 6fff4654be144b74bb03b1adf4dfca63 to VOTER
I20260924 03:15:59.694423 22147 consensus_queue.cc:237] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } }
I20260924 03:15:59.696630 22011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 LEARNER]: Refusing update from remote peer c892c61c6e3f48c0a38a771ea208748d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 03:15:59.697288 21961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 03:15:59.697705 21389 consensus_peers.cc:487] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d -> Peer 163c833a843448b8af4da472b811e5b5 (127.14.227.188:42013): Couldn't send request to peer 163c833a843448b8af4da472b811e5b5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 03:15:59.699484 22148 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6fff4654be144b74bb03b1adf4dfca63 (127.14.227.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.700145 22011 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6fff4654be144b74bb03b1adf4dfca63 (127.14.227.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.703191 22147 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.703228 22148 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.703518 22147 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:59.703518 22148 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
I20260924 03:15:59.704674 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.704926 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:15:59.711966 21965 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 03:15:59.712338 21965 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 03:15:59.712524 21965 tablet_replica.cc:330] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: stopping tablet replica
I20260924 03:15:59.718400 21965 master.cc:552] Master@127.14.227.188:42013 shutdown complete.
I20260924 03:15:59.718495 21965 master_runner.cc:305] Clearing existing system tablet
I20260924 03:15:59.718863 21965 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:59.718977 21965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:59.720964 21965 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:15:59.722371 22153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:59.722647 21965 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:59.722939 21965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:15:59.723034 21965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
uuid: "163c833a843448b8af4da472b811e5b5"
format_stamp: "Formatted at 2026-09-24 03:15:59 on dist-test-slave-vd0p"
server_key: "9793cbd6f4a0730854434c6b68d9237c"
server_key_iv: "60b2267534a4ae497365b462caa92e72"
server_key_version: "encryptionkey@0"
I20260924 03:15:59.726653 21965 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:15:59.731132 21965 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 03:15:59.731287 21965 log.cc:1196] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 03:15:59.731734 21965 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Deleting consensus metadata
I20260924 03:15:59.731962 21965 master_runner.cc:315] Copying system tablet from 127.14.227.190:43857
I20260924 03:15:59.739511 21965 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: tablet copy: Beginning tablet copy session from remote peer at address 127.14.227.190:43857
I20260924 03:15:59.754318 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.740282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.14.227.190:43857)
0924 03:15:59.740556 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 03:15:59.740565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:15:59.742551 (+ 1986us) server_negotiation.cc:407] Connection header received
0924 03:15:59.742839 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:15:59.742844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:15:59.742919 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:15:59.743023 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:15:59.743867 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.744906 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.746173 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.746591 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.746704 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.749052 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:15:59.749058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:15:59.749063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:15:59.749116 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:15:59.751754 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.752362 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.752368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.752372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.752502 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:15:59.752860 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:15:59.752867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:15:59.752870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:15:59.753126 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:15:59.753307 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 03:15:59.753651 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 03:15:59.753875 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 03:15:59.754336 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:15:59.740282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:58494)
0924 03:15:59.742359 (+ 2077us) negotiation.cc:107] Waiting for socket to connect
0924 03:15:59.742379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:15:59.742665 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:15:59.743048 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:15:59.743053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:15:59.743551 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:15:59.743721 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.743729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.745034 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.745039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.746011 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:15:59.746025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:15:59.746758 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:15:59.746765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:15:59.746951 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:15:59.747568 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:15:59.747584 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:15:59.748849 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:15:59.751887 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.751894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.751898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.752245 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.752613 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:15:59.752617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:15:59.752621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:15:59.752758 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:15:59.753241 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:15:59.753246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:15:59.753436 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 03:15:59.753573 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 03:15:59.753881 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1953,"thread_start_us":1818,"threads_started":1}
I20260924 03:15:59.755182 21424 tablet_copy_service.cc:140] P c892c61c6e3f48c0a38a771ea208748d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 163c833a843448b8af4da472b811e5b5 ({username='oryx', principal='oryx/127.14.227.188@KRBTEST.COM'} at 127.0.0.1:58494)
I20260924 03:15:59.755359 21424 tablet_copy_service.cc:161] P c892c61c6e3f48c0a38a771ea208748d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 163c833a843448b8af4da472b811e5b5 at {username='oryx', principal='oryx/127.14.227.188@KRBTEST.COM'} at 127.0.0.1:58494: session id = 163c833a843448b8af4da472b811e5b5-00000000000000000000000000000000
I20260924 03:15:59.757591 21424 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 03:15:59.759253 21965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 03:15:59.763974 21965 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: tablet copy: Starting download of 0 data blocks...
I20260924 03:15:59.764267 21965 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: tablet copy: Starting download of 1 WAL segments...
I20260924 03:15:59.766255 21965 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 03:15:59.769153 21424 tablet_copy_service.cc:342] P c892c61c6e3f48c0a38a771ea208748d: Request end of tablet copy session 163c833a843448b8af4da472b811e5b5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.227.188@KRBTEST.COM'} at 127.0.0.1:58494
I20260924 03:15:59.769371 21424 tablet_copy_service.cc:433] P c892c61c6e3f48c0a38a771ea208748d: ending tablet copy session 163c833a843448b8af4da472b811e5b5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 163c833a843448b8af4da472b811e5b5
I20260924 03:15:59.778429 21965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:15:59.781137 22160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:59.786473 22161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:15:59.793260 22163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:15:59.793538 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 21465
I20260924 03:15:59.793916 21965 server_base.cc:1034] running on GCE node
I20260924 03:15:59.794288 21965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:15:59.794889 21965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:15:59.796077 21965 hybrid_clock.cc:648] HybridClock initialized: now 1790219759796039 us; error 30 us; skew 500 ppm
I20260924 03:15:59.796324 21965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:15:59 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219759, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:15:59.798275 21965 init.cc:377] Logged in from keytab as oryx/127.14.227.188@KRBTEST.COM (short username oryx)
I20260924 03:15:59.802981 21965 webserver.cc:460] Webserver started at http://127.14.227.188:43843/ using document root <none> and password file <none>
I20260924 03:15:59.803357 21965 fs_manager.cc:359] Metadata directory not provided
I20260924 03:15:59.803471 21965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:15:59.806743 21965 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 03:15:59.808760 22170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:15:59.809286 21965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:15:59.809412 21965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
uuid: "163c833a843448b8af4da472b811e5b5"
format_stamp: "Formatted at 2026-09-24 03:15:59 on dist-test-slave-vd0p"
server_key: "9793cbd6f4a0730854434c6b68d9237c"
server_key_iv: "60b2267534a4ae497365b462caa92e72"
server_key_version: "encryptionkey@0"
I20260924 03:15:59.809589 21965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:15:59.817888 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:40295
--local_ip_for_outbound_sockets=127.14.227.129
--tserver_master_addrs=127.14.227.190:43857
--webserver_port=46611
--webserver_interface=127.14.227.129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab
--principal=oryx/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787,127.14.227.188:42013 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:15:59.838578 21965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:15:59.842535 21965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:15:59.879456 21965 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.188:42013
I20260924 03:15:59.879498 22222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.188:42013 every 8 connection(s)
I20260924 03:15:59.880913 21965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
I20260924 03:15:59.886080 22223 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:15:59.889734 22223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Bootstrap starting.
I20260924 03:15:59.907285 22223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:15:59.907922 22223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Bootstrap complete.
I20260924 03:15:59.908993 22223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } }
I20260924 03:15:59.909277 22223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163c833a843448b8af4da472b811e5b5, State: Initialized, Role: LEARNER
I20260924 03:15:59.909528 22223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } }
I20260924 03:15:59.910322 22226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:15:59.910648 22223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:15:59.910888 22226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: This master's current role is: LEARNER
I20260924 03:15:59.921691 22237 catalog_manager.cc:1259] Loaded cluster ID: fb31b95fcf71478bbfc2a060611c62e9
I20260924 03:15:59.921784 22237 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: loading cluster ID for follower catalog manager: success
I20260924 03:15:59.924070 22237 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: acquiring CA information for follower catalog manager: success
I20260924 03:15:59.925654 22237 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:16:00.081341 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.072459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:42013 (local address 127.0.0.1:35236)
0924 03:16:00.072770 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:00.072794 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:00.072978 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:00.074201 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:00.074206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:00.074240 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:00.074441 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.074455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.076853 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.076859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.078812 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.078829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.080518 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.080525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.080727 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.080777 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 03:16:00.080888 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:16:00.081045 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260924 03:16:00.081432 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.072600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.14.227.188:42013)
0924 03:16:00.073666 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.073676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.073702 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:16:00.073799 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.073807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.073924 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.074163 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:00.074631 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.076659 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.078983 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.080343 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.080532 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.080709 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.080923 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.081058 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":674,"threads_started":1}
I20260924 03:16:00.083190 22198 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 03:16:00.084035 22226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c892c61c6e3f48c0a38a771ea208748d. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:16:00.084336 22226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: This master's current role is: LEARNER
W20260924 03:16:00.199751 22175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:00.200067 22175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:00.200232 22175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:00.207659 22175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:00.207787 22175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:00.207966 22175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:16:00.217902 22175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab
--principal=oryx/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:40295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=46611
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787,127.14.227.188:42013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:00.219175 22175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:00.220413 22175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:00.233772 22243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:00.233842 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:00.234020 22246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:00.234681 22175 server_base.cc:1034] running on GCE node
I20260924 03:16:00.235376 22175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:00.236390 22175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:00.237607 22175 hybrid_clock.cc:648] HybridClock initialized: now 1790219760237541 us; error 46 us; skew 500 ppm
I20260924 03:16:00.237928 22175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:00.241438 22175 init.cc:377] Logged in from keytab as oryx/127.14.227.129@KRBTEST.COM (short username oryx)
I20260924 03:16:00.242879 22175 webserver.cc:460] Webserver started at http://127.14.227.129:46611/ using document root <none> and password file <none>
I20260924 03:16:00.243664 22175 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:00.243840 22175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:00.249171 22175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 03:16:00.251724 22253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:00.252982 22175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:16:00.253149 22175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "98785d9ec789413a801ce9bdff5ced2c"
format_stamp: "Formatted at 2026-09-24 03:15:57 on dist-test-slave-vd0p"
server_key: "18f774892281be7180c1708a54f7f487"
server_key_iv: "92ab0e650ad00a294c72d8876af63008"
server_key_version: "encryptionkey@0"
I20260924 03:16:00.254129 22175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:00.292685 22175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:00.297087 22175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:00.297369 22175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:00.298374 22175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:00.299729 22175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:00.299866 22175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:00.300000 22175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:00.300089 22175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:00.337065 22175 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:40295
I20260924 03:16:00.337148 22365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:40295 every 8 connection(s)
I20260924 03:16:00.338595 22175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:16:00.346637 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 22175
I20260924 03:16:00.347064 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 21599
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.189@KRBTEST.COM
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.130@KRBTEST.COM
I20260924 03:16:00.369074 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.341760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:42905 (local address 127.14.227.189:44787)
0924 03:16:00.350322 (+ 8562us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.350327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.350347 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:16:00.350397 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.350401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.350459 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.350608 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:00.352229 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.353314 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.355891 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.356491 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.357440 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.361795 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:00.361800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:00.361804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:00.361833 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:00.364809 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.365771 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.365777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.365780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.365926 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.366555 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.366559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.366562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.366912 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:00.368258 (+ 1346us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.368397 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.368614 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8452,"thread_start_us":8358,"threads_started":1}
I20260924 03:16:00.369903 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.342470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.14.227.129:42905)
0924 03:16:00.344159 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:00.344218 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:00.347732 (+ 3514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:00.350889 (+ 3157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:00.350898 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:00.351409 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:00.351972 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.351989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.353516 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.353522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.355679 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.355693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.356743 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.356754 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.357053 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.357889 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:00.357920 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:00.361576 (+ 3656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:00.364958 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.364967 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.364981 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.365601 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.366086 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.366094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.366097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.366397 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.367065 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:00.367076 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:00.367467 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 03:16:00.367779 (+ 312us) client_negotiation.cc:240] Negotiation successful
0924 03:16:00.368332 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":477,"threads_started":1}
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.188@KRBTEST.COM
I20260924 03:16:00.372082 22368 heartbeater.cc:347] Connected to a master server at 127.14.227.189:44787
I20260924 03:16:00.372412 22368 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:00.373346 22368 heartbeater.cc:511] Master 127.14.227.189:44787 requested a full tablet report, sending...
I20260924 03:16:00.374558 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.348107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.14.227.130:41757)
0924 03:16:00.348868 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.348880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.352084 (+ 3204us) server_negotiation.cc:407] Connection header received
0924 03:16:00.352265 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.352274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.352501 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.352660 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:00.354893 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.356426 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.358722 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.359574 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.359687 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.366834 (+ 7147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:00.366848 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:00.366863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:00.366900 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:00.371111 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.371878 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.371887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.371893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.372042 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.372605 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.372610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.372613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.373025 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:00.373282 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.373890 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.374171 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":358,"threads_started":1}
I20260924 03:16:00.375677 22001 ts_manager.cc:194] Registered new tserver with Master: 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129:40295)
I20260924 03:16:00.378435 21722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:00.380265 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.342470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.129:52295)
0924 03:16:00.359649 (+ 17179us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:00.359677 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:00.359823 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:00.360234 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:00.360239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:00.361996 (+ 1757us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:00.362275 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.362289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.364386 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.364394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.366478 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.366492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.367550 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.367555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.367711 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.368304 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:00.368333 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:00.372175 (+ 3842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:00.377134 (+ 4959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.377137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.377140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.377595 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.378003 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.378009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.378013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.378174 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.379079 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:00.379086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:00.379244 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 03:16:00.379376 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 03:16:00.379712 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16977,"mutex_wait_us":445,"thread_start_us":17749,"threads_started":1}
I20260924 03:16:00.380407 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.341722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:52295 (local address 127.14.227.190:43857)
0924 03:16:00.358991 (+ 17269us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.359001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.359930 (+ 929us) server_negotiation.cc:407] Connection header received
0924 03:16:00.360002 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.360008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.360079 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.360185 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:00.362449 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.363575 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.366647 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.367217 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.367334 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.372357 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:00.372363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:00.372368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:00.372408 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:00.377014 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.377722 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.377727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.377730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.377877 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.378298 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.378305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.378309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.378692 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:00.378903 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.379570 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.379885 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17117,"thread_start_us":17283,"threads_started":1}
I20260924 03:16:00.381568 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.344687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:57233 (local address 127.14.227.188:42013)
0924 03:16:00.344910 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.344918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.360908 (+ 15990us) server_negotiation.cc:407] Connection header received
0924 03:16:00.361169 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.361175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.361259 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.361357 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:00.363019 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.364225 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.367301 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.367919 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.368032 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.373135 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:00.373142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:00.373149 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:00.373190 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:00.377122 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.377905 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.377912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.377918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.378070 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.378467 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.378472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.378475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.378841 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:00.379008 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.380945 (+ 1937us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.381181 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:16:00.381868 22367 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43857
I20260924 03:16:00.382017 22367 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:00.382476 22367 heartbeater.cc:511] Master 127.14.227.190:43857 requested a full tablet report, sending...
I20260924 03:16:00.382881 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.344269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:42013 (local address 127.14.227.129:57233)
0924 03:16:00.360797 (+ 16528us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:00.360819 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:00.360998 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:00.361496 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:00.361500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:00.362515 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:00.362834 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.362847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.365141 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.365147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.367144 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.367156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.368067 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.368073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.369450 (+ 1377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.370279 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:00.370304 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:00.372913 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:00.377220 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.377225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.377228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.377765 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.378178 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.378182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.378185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.378357 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.380501 (+ 2144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:00.380506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:00.380649 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 03:16:00.381928 (+ 1279us) client_negotiation.cc:240] Negotiation successful
0924 03:16:00.382223 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16399,"mutex_wait_us":1,"thread_start_us":15019,"threads_started":1}
I20260924 03:16:00.383644 21404 ts_manager.cc:194] Re-registered known tserver with Master: 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129:40295)
I20260924 03:16:00.383876 22366 heartbeater.cc:347] Connected to a master server at 127.14.227.188:42013
I20260924 03:16:00.383997 22366 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:00.384369 22366 heartbeater.cc:511] Master 127.14.227.188:42013 requested a full tablet report, sending...
I20260924 03:16:00.384716 21404 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:52295
I20260924 03:16:00.386117 22188 ts_manager.cc:194] Registered new tserver with Master: 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129:40295)
I20260924 03:16:00.475605 22148 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: NON_VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: true } } }
I20260924 03:16:00.476035 22148 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:00.479516 22379 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: attempting to promote NON_VOTER 163c833a843448b8af4da472b811e5b5 to VOTER
I20260924 03:16:00.480119 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 21599
I20260924 03:16:00.480357 22379 consensus_queue.cc:237] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } }
I20260924 03:16:00.483165 22198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 LEARNER]: Refusing update from remote peer c892c61c6e3f48c0a38a771ea208748d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 03:16:00.483191 22011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Refusing update from remote peer c892c61c6e3f48c0a38a771ea208748d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 03:16:00.483889 22379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 03:16:00.484344 22378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 03:16:00.487968 22379 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 163c833a843448b8af4da472b811e5b5 (127.14.227.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:00.489287 22198 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 163c833a843448b8af4da472b811e5b5 (127.14.227.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:00.492268 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:00.492595 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:00.492820 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:00.492828 22226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:00.493053 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:00.493091 22226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:16:00.494109 22011 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 163c833a843448b8af4da472b811e5b5 (127.14.227.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:00.498483 22378 sys_catalog.cc:455] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:00.498836 22378 sys_catalog.cc:458] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:00.499455 22383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c892c61c6e3f48c0a38a771ea208748d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:00.499951 22383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:16:00.500938 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:41757
--local_ip_for_outbound_sockets=127.14.227.130
--tserver_master_addrs=127.14.227.190:43857
--webserver_port=39717
--webserver_interface=127.14.227.130
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab
--principal=oryx/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787,127.14.227.188:42013 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:00.823076 22384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:00.823449 22384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:00.823632 22384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:00.832348 22384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:00.832499 22384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:00.832683 22384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:16:00.843684 22384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab
--principal=oryx/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:41757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=39717
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787,127.14.227.188:42013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:00.845031 22384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:00.846436 22384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:00.859957 22390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:00.859910 22392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:00.859995 22389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:00.861032 22384 server_base.cc:1034] running on GCE node
I20260924 03:16:00.861536 22384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:00.862418 22384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:00.863636 22384 hybrid_clock.cc:648] HybridClock initialized: now 1790219760863575 us; error 51 us; skew 500 ppm
I20260924 03:16:00.863931 22384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:00.866838 22384 init.cc:377] Logged in from keytab as oryx/127.14.227.130@KRBTEST.COM (short username oryx)
I20260924 03:16:00.868119 22384 webserver.cc:460] Webserver started at http://127.14.227.130:39717/ using document root <none> and password file <none>
I20260924 03:16:00.868789 22384 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:00.868920 22384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:00.873907 22384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:16:00.876626 22399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:00.877936 22384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:16:00.878130 22384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "96be3ec41d44425a96b119bb6978e3ff"
format_stamp: "Formatted at 2026-09-24 03:15:57 on dist-test-slave-vd0p"
server_key: "cde951a968107675659a841b50579f1a"
server_key_iv: "c1bbc3054a9b15d4e118389b22290bd1"
server_key_version: "encryptionkey@0"
I20260924 03:16:00.879099 22384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:00.894335 22384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:00.897816 22384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:00.898102 22384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:00.898937 22384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:00.900101 22384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:00.900228 22384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:00.900342 22384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:00.900427 22384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:00.932271 22384 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:41757
I20260924 03:16:00.932374 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:41757 every 8 connection(s)
I20260924 03:16:00.933656 22384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:16:00.940510 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 22384
I20260924 03:16:00.940869 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 21732
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.188@KRBTEST.COM
I20260924 03:16:00.964525 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.937589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:43789 (local address 127.14.227.190:43857)
0924 03:16:00.938066 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.938074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.938099 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:16:00.941098 (+ 2999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.941107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.941228 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.941354 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:00.944754 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.946258 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.949458 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.950365 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.950492 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.955627 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:00.955636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:00.955644 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:00.955695 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:00.959973 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.961487 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.961492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.961496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.961642 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.962192 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.962199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.962203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.962500 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:00.962645 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.963926 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.964147 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":700,"threads_started":1}
I20260924 03:16:00.965260 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.937064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.130:43789)
0924 03:16:00.937968 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:00.938016 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:00.940387 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:00.942015 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:00.942030 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:00.943004 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:00.944520 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.944553 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.946498 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.946503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.949258 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.949277 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.950925 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.950935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.951319 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.952222 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:00.952268 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:00.955369 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:00.960169 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.960177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.960191 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.961366 (+ 1175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.961758 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.961763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.961767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.962047 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.962655 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:00.962662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:00.963032 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 03:16:00.963344 (+ 312us) client_negotiation.cc:240] Negotiation successful
0924 03:16:00.963827 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":1177,"threads_started":1}
I20260924 03:16:00.966789 22513 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43857
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219760, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.189@KRBTEST.COM
I20260924 03:16:00.967119 22513 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:00.968115 22513 heartbeater.cc:511] Master 127.14.227.190:43857 requested a full tablet report, sending...
I20260924 03:16:00.969892 21404 ts_manager.cc:194] Re-registered known tserver with Master: 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757)
I20260924 03:16:00.970417 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.938062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:42013 (local address 127.14.227.130:58219)
0924 03:16:00.938783 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:00.938802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:00.941514 (+ 2712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:00.952159 (+ 10645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:00.952163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:00.952833 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:00.953127 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.953139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.954605 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.954610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.956173 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.956185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.957158 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.957165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.957346 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.957982 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:00.958009 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:00.960810 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:00.967620 (+ 6810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.967625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.967628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.968232 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.968687 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.968692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.968695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.968831 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.969499 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:00.969505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:00.969672 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 03:16:00.969799 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:16:00.970071 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":1230,"threads_started":1}
Sep 24 03:16:00 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.131@KRBTEST.COM
I20260924 03:16:00.971045 21404 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:43789
I20260924 03:16:00.971324 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.936273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:58219 (local address 127.14.227.188:42013)
0924 03:16:00.951707 (+ 15434us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.951714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.951733 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:16:00.951790 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.951795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.951866 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.952108 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:00.953287 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.954446 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.956330 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.956950 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.957665 (+ 715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.960995 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:00.961001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:00.961005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:00.961042 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:00.967472 (+ 6430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.968393 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.968401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.968405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.968542 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.968993 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.968998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.969001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.969323 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:00.970680 (+ 1357us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.970798 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.970987 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15271,"thread_start_us":15678,"threads_started":1}
I20260924 03:16:00.972374 22512 heartbeater.cc:347] Connected to a master server at 127.14.227.188:42013
I20260924 03:16:00.972508 22512 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:00.973410 22512 heartbeater.cc:511] Master 127.14.227.188:42013 requested a full tablet report, sending...
I20260924 03:16:00.974525 22188 ts_manager.cc:194] Registered new tserver with Master: 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757)
I20260924 03:16:00.974820 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.940010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:58437 (local address 127.14.227.189:44787)
0924 03:16:00.952034 (+ 12024us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.952038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.958394 (+ 6356us) server_negotiation.cc:407] Connection header received
0924 03:16:00.958578 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.958584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.958684 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.958799 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:00.960078 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.960857 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.962704 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.963371 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.963859 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.967925 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:00.967933 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:00.967937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:00.967981 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:00.971667 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.972457 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.972462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.972465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.972602 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.973206 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.973211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.973215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.973567 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:00.973709 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.974244 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.974455 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11920,"thread_start_us":11820,"threads_started":1}
I20260924 03:16:00.975603 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.939869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.14.227.130:58437)
0924 03:16:00.958287 (+ 18418us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:00.958308 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:00.958432 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:00.958987 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:00.958991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:00.959588 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:00.959904 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.959917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.961053 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.961057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.962559 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:00.962573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.964165 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.964171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:00.964368 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.964927 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:00.964950 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:00.967699 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:00.971844 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.971848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.971851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.972318 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.972739 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:00.972746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:00.972751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:00.973063 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:00.973800 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:00.973804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:00.973982 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 03:16:00.975065 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0924 03:16:00.975356 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18284,"thread_start_us":18106,"threads_started":1}
I20260924 03:16:00.976411 22514 heartbeater.cc:347] Connected to a master server at 127.14.227.189:44787
I20260924 03:16:00.976555 22514 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:00.976945 22514 heartbeater.cc:511] Master 127.14.227.189:44787 requested a full tablet report, sending...
I20260924 03:16:00.977897 22001 ts_manager.cc:194] Registered new tserver with Master: 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757)
I20260924 03:16:00.978691 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:00.942326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.14.227.131:38741)
0924 03:16:00.954206 (+ 11880us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:00.954214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:00.956030 (+ 1816us) server_negotiation.cc:407] Connection header received
0924 03:16:00.956101 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:00.956109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:00.956380 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:00.956510 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:00.959684 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.961863 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.963508 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:00.964302 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:00.964426 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:00.971852 (+ 7426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:00.971866 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:00.971883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:00.971922 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:00.975846 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.976526 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.976533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.976536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.976723 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:00.977078 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:00.977083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:00.977086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:00.977487 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:00.977772 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:00.978100 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 03:16:00.978352 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11691,"thread_start_us":11505,"threads_started":1}
I20260924 03:16:00.979738 21855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:01.063112 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 21732
I20260924 03:16:01.078661 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:38741
--local_ip_for_outbound_sockets=127.14.227.131
--tserver_master_addrs=127.14.227.190:43857
--webserver_port=38683
--webserver_interface=127.14.227.131
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab
--principal=oryx/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787,127.14.227.188:42013 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:16:01.388494 22367 heartbeater.cc:503] Master 127.14.227.190:43857 was elected leader, sending a full tablet report...
W20260924 03:16:01.400301 22525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:01.400553 22525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:01.400662 22525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:01.407488 22525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:01.407573 22525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:01.407706 22525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:16:01.417872 22525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab
--principal=oryx/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:38741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=38683
--tserver_master_addrs=127.14.227.190:43857,127.14.227.189:44787,127.14.227.188:42013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:01.419209 22525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:01.420614 22525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:01.434291 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:01.434327 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:01.434374 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:01.435168 22525 server_base.cc:1034] running on GCE node
I20260924 03:16:01.435770 22525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:01.436748 22525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:01.437994 22525 hybrid_clock.cc:648] HybridClock initialized: now 1790219761437923 us; error 40 us; skew 500 ppm
I20260924 03:16:01.438320 22525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:01 dist-test-slave-vd0p krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219761, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:01.441553 22525 init.cc:377] Logged in from keytab as oryx/127.14.227.131@KRBTEST.COM (short username oryx)
I20260924 03:16:01.443020 22525 webserver.cc:460] Webserver started at http://127.14.227.131:38683/ using document root <none> and password file <none>
I20260924 03:16:01.443720 22525 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:01.443868 22525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:01.449194 22525 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 03:16:01.451766 22540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:01.453004 22525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:16:01.453182 22525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "bb775656b7d9489fa30ee0eeab6484bb"
format_stamp: "Formatted at 2026-09-24 03:15:58 on dist-test-slave-vd0p"
server_key: "9dc4bdc11e92cf90ce7dfcf58b334731"
server_key_iv: "0ed754dc1562126a19ff543d84f37db5"
server_key_version: "encryptionkey@0"
I20260924 03:16:01.454207 22525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:01.474524 22525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:01.478883 22525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:01.479171 22525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:01.480100 22525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:01.481392 22525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:01.481506 22525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:01.481631 22525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:01.481745 22525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:01.513700 22525 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:38741
I20260924 03:16:01.513731 22652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:38741 every 8 connection(s)
I20260924 03:16:01.515264 22525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:16:01.517078 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 22525
Sep 24 03:16:01 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219761, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
Sep 24 03:16:01 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219761, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.189@KRBTEST.COM
Sep 24 03:16:01 dist-test-slave-vd0p krb5kdc[21359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219761, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.188@KRBTEST.COM
I20260924 03:16:01.563941 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.519041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:58885 (local address 127.14.227.190:43857)
0924 03:16:01.530209 (+ 11168us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.530219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.535410 (+ 5191us) server_negotiation.cc:407] Connection header received
0924 03:16:01.535580 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.535588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.535689 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.535813 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:01.538031 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.539176 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.541061 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.541740 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.541871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.550583 (+ 8712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:01.550601 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:01.550610 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:01.550675 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:01.556528 (+ 5853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.559319 (+ 2791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.559329 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.559335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.559544 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.560868 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.560877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.560883 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.561412 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:01.561618 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.562955 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.563306 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10994,"thread_start_us":11159,"threads_started":1}
I20260924 03:16:01.565267 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.518801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.131:58885)
0924 03:16:01.535304 (+ 16503us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:01.535319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:01.535461 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:01.536140 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:01.536149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:01.537020 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:01.537842 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.537869 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.539577 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.539580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.540802 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.540812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.542724 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.542731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.543034 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.543962 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:01.544005 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:01.550189 (+ 6184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:01.558385 (+ 8196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:01.558397 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:01.558418 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:01.559156 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:01.560413 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:01.560420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:01.560427 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:01.560712 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:01.561898 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:01.561910 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:01.562461 (+ 551us) client_negotiation.cc:769] Sending connection context
0924 03:16:01.562800 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 03:16:01.563495 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16388,"mutex_wait_us":368,"thread_start_us":16632,"threads_started":1}
I20260924 03:16:01.567150 22654 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43857
I20260924 03:16:01.567469 22654 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:01.567860 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.531705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:44475 (local address 127.14.227.188:42013)
0924 03:16:01.543131 (+ 11426us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.543140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.543163 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:16:01.543248 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.543255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.543366 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.544369 (+ 1003us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:01.545084 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.546391 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.548380 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.548979 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.549071 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.553630 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:01.553639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:01.553646 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:01.553695 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:01.558079 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.560096 (+ 2017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.560104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.560110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.560272 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.561191 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.561199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.561206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.561649 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:01.561797 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.567213 (+ 5416us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.567428 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11277,"thread_start_us":11761,"threads_started":1}
I20260924 03:16:01.567984 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.518922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:43219 (local address 127.14.227.189:44787)
0924 03:16:01.538388 (+ 19466us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.538395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.538413 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:16:01.538469 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.538474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.538545 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.541125 (+ 2580us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:01.541308 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.542358 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.544467 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.545202 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.546038 (+ 836us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.549276 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:01.549287 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:01.549296 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:01.549351 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:01.560715 (+ 11364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.562724 (+ 2009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.562733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.562739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.564347 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.565019 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.565028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.565034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.565525 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:01.567097 (+ 1572us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.567243 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.567472 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19332,"thread_start_us":19209,"threads_started":1}
I20260924 03:16:01.568424 22654 heartbeater.cc:511] Master 127.14.227.190:43857 requested a full tablet report, sending...
I20260924 03:16:01.568987 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.537271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:42013 (local address 127.14.227.131:44475)
0924 03:16:01.537803 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:01.537821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:01.537922 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:01.543672 (+ 5750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:01.543685 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:01.544523 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:01.544871 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.544888 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.546588 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.546593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.548221 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.548236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.549382 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.549392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.549654 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.550434 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:01.550462 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:01.553361 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:01.559311 (+ 5950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:01.559317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:01.559324 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:01.559921 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:01.560867 (+ 946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:01.560874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:01.560880 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:01.561049 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:01.566007 (+ 4958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:01.566014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:01.566214 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 03:16:01.566392 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 03:16:01.568143 (+ 1751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":429,"threads_started":1}
I20260924 03:16:01.569305 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.518800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.14.227.131:43219)
0924 03:16:01.531072 (+ 12272us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:01.531131 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:01.533939 (+ 2808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:01.538712 (+ 4773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:01.538718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:01.539121 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:01.539423 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.539451 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.542528 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.542535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.544234 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.544251 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.545424 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.545431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.545638 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.546297 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:01.546337 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:01.549005 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:01.561772 (+ 12767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:01.561780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:01.561787 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:01.562499 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:01.564571 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:01.564580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:01.564587 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:01.564828 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:01.565687 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:01.565696 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:01.566706 (+ 1010us) client_negotiation.cc:769] Sending connection context
0924 03:16:01.566859 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 03:16:01.568317 (+ 1458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11889,"mutex_wait_us":1121,"thread_start_us":11583,"threads_started":1}
I20260924 03:16:01.570242 22655 heartbeater.cc:347] Connected to a master server at 127.14.227.189:44787
I20260924 03:16:01.570380 22655 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:01.570887 22655 heartbeater.cc:511] Master 127.14.227.189:44787 requested a full tablet report, sending...
I20260924 03:16:01.571408 22653 heartbeater.cc:347] Connected to a master server at 127.14.227.188:42013
I20260924 03:16:01.571396 21404 ts_manager.cc:194] Re-registered known tserver with Master: bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131:38741)
I20260924 03:16:01.571504 22653 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:01.571878 22653 heartbeater.cc:511] Master 127.14.227.188:42013 requested a full tablet report, sending...
I20260924 03:16:01.572768 22001 ts_manager.cc:194] Registered new tserver with Master: bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131:38741)
I20260924 03:16:01.572990 21404 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:58885
I20260924 03:16:01.574035 22188 ts_manager.cc:194] Registered new tserver with Master: bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131:38741)
I20260924 03:16:01.587785 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.548990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.14.227.190:43857)
0924 03:16:01.556039 (+ 7049us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.556047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.569744 (+ 13697us) server_negotiation.cc:407] Connection header received
0924 03:16:01.569799 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.569804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.569879 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.569995 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:01.571932 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.573038 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.574874 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.575505 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.575639 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.580189 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:01.580196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:01.580201 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:01.580240 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:01.583882 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.584757 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.584764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.584767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.584922 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.585603 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.585610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.585614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.586016 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:01.586270 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.586690 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.586915 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6817,"thread_start_us":5683,"threads_started":1}
I20260924 03:16:01.589564 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.553268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.14.227.189:44787)
0924 03:16:01.566490 (+ 13222us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.566495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.567294 (+ 799us) server_negotiation.cc:407] Connection header received
0924 03:16:01.567346 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.567354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.567424 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.567515 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:01.575882 (+ 8367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.576926 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.579299 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.579761 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.579902 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.583288 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:01.583296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:01.583303 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:01.583342 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:01.586563 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.587299 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.587304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.587308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.587459 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.587821 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.587825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.587829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.588177 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:01.588411 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.588874 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.589089 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13114,"thread_start_us":13931,"threads_started":1}
I20260924 03:16:01.593993 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.569342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.14.227.188:42013)
0924 03:16:01.570938 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.570944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.578089 (+ 7145us) server_negotiation.cc:407] Connection header received
0924 03:16:01.578257 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.578262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.578354 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.578467 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:01.580482 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.581334 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.583128 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.583594 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.583785 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.586983 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:01.586990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:01.586995 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:01.587031 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:01.591484 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.592119 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.592125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.592129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.592306 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.592698 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.592701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.592703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.592969 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:01.593134 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.593477 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.593633 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492}
I20260924 03:16:01.592018 21404 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:16:01.599866 21404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:16:01.631443 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.618240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38741 (local address 127.0.0.1:51272)
0924 03:16:01.622546 (+ 4306us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:01.622588 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:01.622728 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:01.624202 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:01.624206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:01.624227 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:01.624491 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.624498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.627032 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.627036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.628800 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.628812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.630682 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.630689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.630857 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.630887 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:16:01.630987 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:16:01.631105 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4145,"thread_start_us":6442,"threads_started":1}
I20260924 03:16:01.631466 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.618467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.14.227.131:38741)
0924 03:16:01.623467 (+ 5000us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.623479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.623521 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:16:01.623640 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.623652 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.623985 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.624231 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:01.624680 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.626559 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.628944 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.630498 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.630656 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.630825 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.631029 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.631181 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4788,"thread_start_us":4568,"threads_started":1}
I20260924 03:16:01.634976 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.618949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.14.227.129:40295)
0924 03:16:01.619851 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.619864 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.625044 (+ 5180us) server_negotiation.cc:407] Connection header received
0924 03:16:01.625381 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.625393 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.625674 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.625845 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:01.626967 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.628883 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.632126 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.633556 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.633733 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.633899 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.634465 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.634663 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":643,"threads_started":1}
I20260924 03:16:01.635339 22588 tablet_service.cc:1467] Processing CreateTablet for tablet 887c0d2676e74194a6cf272b59071bca (DEFAULT_TABLE table=test-table [id=104111a482ed4582903209911bd429b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:01.635810 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.618761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:40295 (local address 127.0.0.1:56670)
0924 03:16:01.624940 (+ 6179us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:01.624955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:01.625165 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:01.626192 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:01.626199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:01.626232 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:01.626749 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.626761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.629101 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.629109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.631996 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.632007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.634078 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.634085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.634256 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.634296 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:16:01.635254 (+ 958us) client_negotiation.cc:240] Negotiation successful
0924 03:16:01.635392 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6039,"mutex_wait_us":1,"thread_start_us":3442,"threads_started":1}
I20260924 03:16:01.636708 22588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887c0d2676e74194a6cf272b59071bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:01.639310 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.625030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:41757 (local address 127.0.0.1:52448)
0924 03:16:01.630403 (+ 5373us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:01.630421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:01.630538 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:01.632227 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:01.632232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:01.632256 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:01.632513 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.632521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.634657 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.634662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.636863 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.636875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.638542 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.638551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.638771 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.638803 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:16:01.638888 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:16:01.639004 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5247,"thread_start_us":5065,"threads_started":1}
I20260924 03:16:01.639729 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.625282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.14.227.130:41757)
0924 03:16:01.627035 (+ 1753us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.627042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.630885 (+ 3843us) server_negotiation.cc:407] Connection header received
0924 03:16:01.631814 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.631822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.632067 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.632249 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:01.632709 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.634345 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.637005 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.638040 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.638190 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.638340 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.638949 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.639106 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1621,"thread_start_us":1522,"threads_started":1}
I20260924 03:16:01.639379 22301 tablet_service.cc:1467] Processing CreateTablet for tablet 887c0d2676e74194a6cf272b59071bca (DEFAULT_TABLE table=test-table [id=104111a482ed4582903209911bd429b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:01.640641 22301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887c0d2676e74194a6cf272b59071bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:01.642822 22447 tablet_service.cc:1467] Processing CreateTablet for tablet 887c0d2676e74194a6cf272b59071bca (DEFAULT_TABLE table=test-table [id=104111a482ed4582903209911bd429b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:01.644073 22447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887c0d2676e74194a6cf272b59071bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:01.652751 22689 tablet_bootstrap.cc:492] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Bootstrap starting.
I20260924 03:16:01.655392 22690 tablet_bootstrap.cc:492] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Bootstrap starting.
I20260924 03:16:01.658269 22689 tablet_bootstrap.cc:654] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:01.658736 22690 tablet_bootstrap.cc:654] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:01.659063 22691 tablet_bootstrap.cc:492] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Bootstrap starting.
I20260924 03:16:01.659890 22690 log.cc:824] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:01.660216 22689 log.cc:824] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:01.662381 22691 tablet_bootstrap.cc:654] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:01.663029 22690 tablet_bootstrap.cc:492] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: No bootstrap required, opened a new log
I20260924 03:16:01.663264 22689 tablet_bootstrap.cc:492] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: No bootstrap required, opened a new log
I20260924 03:16:01.663302 22690 ts_tablet_manager.cc:1397] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 03:16:01.663522 22689 ts_tablet_manager.cc:1397] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 03:16:01.663559 22691 log.cc:824] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:01.667100 22691 tablet_bootstrap.cc:492] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: No bootstrap required, opened a new log
I20260924 03:16:01.667351 22691 ts_tablet_manager.cc:1397] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260924 03:16:01.671772 22689 raft_consensus.cc:348] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.672122 22689 raft_consensus.cc:374] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:01.672215 22689 raft_consensus.cc:729] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb775656b7d9489fa30ee0eeab6484bb, State: Initialized, Role: FOLLOWER
I20260924 03:16:01.673252 22689 consensus_queue.cc:260] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.673344 22690 raft_consensus.cc:348] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.673727 22690 raft_consensus.cc:374] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:01.673818 22690 raft_consensus.cc:729] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98785d9ec789413a801ce9bdff5ced2c, State: Initialized, Role: FOLLOWER
I20260924 03:16:01.674382 22690 consensus_queue.cc:260] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.674608 22654 heartbeater.cc:503] Master 127.14.227.190:43857 was elected leader, sending a full tablet report...
I20260924 03:16:01.678117 22691 raft_consensus.cc:348] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.678627 22691 raft_consensus.cc:374] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:01.678740 22691 raft_consensus.cc:729] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96be3ec41d44425a96b119bb6978e3ff, State: Initialized, Role: FOLLOWER
I20260924 03:16:01.679095 22689 ts_tablet_manager.cc:1428] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260924 03:16:01.679539 22691 consensus_queue.cc:260] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.680994 22690 ts_tablet_manager.cc:1428] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260924 03:16:01.686496 22513 heartbeater.cc:503] Master 127.14.227.190:43857 was elected leader, sending a full tablet report...
I20260924 03:16:01.687089 22691 ts_tablet_manager.cc:1428] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Time spent starting tablet: real 0.020s user 0.006s sys 0.014s
W20260924 03:16:01.687417 22515 tablet.cc:2369] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:16:01.785329 22656 tablet.cc:2369] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:16:01.786716 22695 raft_consensus.cc:482] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:16:01.787004 22695 raft_consensus.cc:504] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.788570 22695 leader_election.cc:288] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129:40295), 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757)
I20260924 03:16:01.796268 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.789033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:59547 (local address 127.14.227.130:41757)
0924 03:16:01.789236 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.789242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.789262 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:16:01.789405 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.789410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.789479 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.789571 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:01.790090 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.791157 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.793929 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.795178 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.795339 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.795407 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.795896 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.796044 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:16:01.796268 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.788851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:36469 (local address 127.14.227.129:40295)
0924 03:16:01.789033 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.789039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.789237 (+ 198us) server_negotiation.cc:407] Connection header received
0924 03:16:01.789405 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.789410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.789479 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.789571 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:01.790090 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.791154 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.793925 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.795169 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.795339 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.795407 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.795896 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.796016 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:16:01.796363 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.788871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:40295 (local address 127.14.227.131:36469)
0924 03:16:01.789137 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:01.789153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:01.789278 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:01.789598 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:01.789602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:01.789629 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:01.789958 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.789968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.791288 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.791293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.793775 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.793791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.795342 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.795348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.795666 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.795773 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 03:16:01.795862 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 03:16:01.796030 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":369,"spinlock_wait_cycles":1792}
I20260924 03:16:01.796766 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.789001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:41757 (local address 127.14.227.131:59547)
0924 03:16:01.789188 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:01.789201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:01.789278 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:01.789598 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:01.789602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:01.789637 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:01.789956 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.789968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.791288 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.791293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.793776 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:01.793790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.795342 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.795348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:01.795670 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.795771 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 03:16:01.795862 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 03:16:01.796296 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":678,"spinlock_wait_cycles":21504}
I20260924 03:16:01.797324 22321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887c0d2676e74194a6cf272b59071bca" candidate_uuid: "bb775656b7d9489fa30ee0eeab6484bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98785d9ec789413a801ce9bdff5ced2c" is_pre_election: true
I20260924 03:16:01.797719 22467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887c0d2676e74194a6cf272b59071bca" candidate_uuid: "bb775656b7d9489fa30ee0eeab6484bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be3ec41d44425a96b119bb6978e3ff" is_pre_election: true
I20260924 03:16:01.797863 22321 raft_consensus.cc:2454] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb775656b7d9489fa30ee0eeab6484bb in term 0.
I20260924 03:16:01.798220 22467 raft_consensus.cc:2454] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb775656b7d9489fa30ee0eeab6484bb in term 0.
I20260924 03:16:01.798599 22541 leader_election.cc:302] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98785d9ec789413a801ce9bdff5ced2c, bb775656b7d9489fa30ee0eeab6484bb; no voters:
I20260924 03:16:01.799032 22695 raft_consensus.cc:2781] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:16:01.799178 22695 raft_consensus.cc:482] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:16:01.799289 22695 raft_consensus.cc:3037] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:01.801720 22695 raft_consensus.cc:504] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.802345 22695 leader_election.cc:288] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [CANDIDATE]: Term 1 election: Requested vote from peers 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129:40295), 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757)
I20260924 03:16:01.802723 22321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887c0d2676e74194a6cf272b59071bca" candidate_uuid: "bb775656b7d9489fa30ee0eeab6484bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98785d9ec789413a801ce9bdff5ced2c"
I20260924 03:16:01.802825 22467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887c0d2676e74194a6cf272b59071bca" candidate_uuid: "bb775656b7d9489fa30ee0eeab6484bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be3ec41d44425a96b119bb6978e3ff"
I20260924 03:16:01.802884 22321 raft_consensus.cc:3037] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:01.803025 22467 raft_consensus.cc:3037] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:01.805048 22321 raft_consensus.cc:2454] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb775656b7d9489fa30ee0eeab6484bb in term 1.
I20260924 03:16:01.805117 22467 raft_consensus.cc:2454] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb775656b7d9489fa30ee0eeab6484bb in term 1.
I20260924 03:16:01.805476 22541 leader_election.cc:302] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98785d9ec789413a801ce9bdff5ced2c, bb775656b7d9489fa30ee0eeab6484bb; no voters:
I20260924 03:16:01.805751 22695 raft_consensus.cc:2781] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:01.806191 22695 raft_consensus.cc:686] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 1 LEADER]: Becoming Leader. State: Replica: bb775656b7d9489fa30ee0eeab6484bb, State: Running, Role: LEADER
I20260924 03:16:01.806900 22695 consensus_queue.cc:237] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:01.811817 21404 catalog_manager.cc:5495] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb reported cstate change: term changed from 0 to 1, leader changed from <none> to bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131). New cstate: current_term: 1 leader_uuid: "bb775656b7d9489fa30ee0eeab6484bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } health_report { overall_health: UNKNOWN } } }
W20260924 03:16:01.842713 22369 tablet.cc:2369] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:16:01.873667 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.868889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.14.227.131:38741)
0924 03:16:01.869066 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.869071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.869177 (+ 106us) server_negotiation.cc:407] Connection header received
0924 03:16:01.869353 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.869357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.869468 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.869617 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:01.869924 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.870718 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.871935 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.872317 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.872402 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.872798 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:01.873262 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:01.873370 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.873427 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.873480 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:16:01.881366 22467 raft_consensus.cc:1261] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 1 FOLLOWER]: Refusing update from remote peer bb775656b7d9489fa30ee0eeab6484bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:01.881407 22321 raft_consensus.cc:1261] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 1 FOLLOWER]: Refusing update from remote peer bb775656b7d9489fa30ee0eeab6484bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:01.882377 22698 consensus_queue.cc:1035] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:01.883165 22695 consensus_queue.cc:1035] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:01.928270 21404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:01.928514 21404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:01.934368 21404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c892c61c6e3f48c0a38a771ea208748d: Sending DeleteTablet for 3 replicas of tablet 887c0d2676e74194a6cf272b59071bca
I20260924 03:16:01.936121 22447 tablet_service.cc:1514] Processing DeleteTablet for tablet 887c0d2676e74194a6cf272b59071bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:01 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:52448
I20260924 03:16:01.936344 22588 tablet_service.cc:1514] Processing DeleteTablet for tablet 887c0d2676e74194a6cf272b59071bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:01 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:51272
I20260924 03:16:01.937525 22712 tablet_replica.cc:330] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: stopping tablet replica
I20260924 03:16:01.937907 22712 raft_consensus.cc:2229] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:16:01.938570 22712 raft_consensus.cc:2258] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:01.939229 22301 tablet_service.cc:1514] Processing DeleteTablet for tablet 887c0d2676e74194a6cf272b59071bca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:01 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56670
I20260924 03:16:01.939898 22711 tablet_replica.cc:330] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: stopping tablet replica
I20260924 03:16:01.940428 22711 raft_consensus.cc:2229] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:01.941067 22711 raft_consensus.cc:2258] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:01.943718 22711 ts_tablet_manager.cc:1905] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:01.948704 22712 ts_tablet_manager.cc:1905] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:01.950418 22714 tablet_replica.cc:330] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: stopping tablet replica
I20260924 03:16:01.950800 22714 raft_consensus.cc:2229] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:01.951892 22714 raft_consensus.cc:2258] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:01.953125 22711 ts_tablet_manager.cc:1918] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:01.953320 22711 log.cc:1196] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/887c0d2676e74194a6cf272b59071bca
I20260924 03:16:01.953670 22714 ts_tablet_manager.cc:1905] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:01.954066 22711 ts_tablet_manager.cc:1939] T 887c0d2676e74194a6cf272b59071bca P 96be3ec41d44425a96b119bb6978e3ff: Deleting consensus metadata
I20260924 03:16:01.955757 21391 catalog_manager.cc:4841] TS 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757): tablet 887c0d2676e74194a6cf272b59071bca (table test-table [id=104111a482ed4582903209911bd429b3]) successfully deleted
I20260924 03:16:01.957805 22712 ts_tablet_manager.cc:1918] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:01.959714 22714 ts_tablet_manager.cc:1918] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:01.959870 22714 log.cc:1196] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/887c0d2676e74194a6cf272b59071bca
I20260924 03:16:01.960299 22714 ts_tablet_manager.cc:1939] T 887c0d2676e74194a6cf272b59071bca P 98785d9ec789413a801ce9bdff5ced2c: Deleting consensus metadata
I20260924 03:16:01.961649 21388 catalog_manager.cc:4841] TS 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129:40295): tablet 887c0d2676e74194a6cf272b59071bca (table test-table [id=104111a482ed4582903209911bd429b3]) successfully deleted
I20260924 03:16:01.966078 22712 log.cc:1196] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/887c0d2676e74194a6cf272b59071bca
I20260924 03:16:01.966735 22712 ts_tablet_manager.cc:1939] T 887c0d2676e74194a6cf272b59071bca P bb775656b7d9489fa30ee0eeab6484bb: Deleting consensus metadata
I20260924 03:16:01.968211 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 21372
I20260924 03:16:01.969331 21434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:01.984889 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.937555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.14.227.189:44787)
0924 03:16:01.937752 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.937758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.966547 (+ 28789us) server_negotiation.cc:407] Connection header received
0924 03:16:01.966863 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.966871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.966955 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.967029 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:01.970140 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.971243 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.973064 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.973727 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.973816 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.977768 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:01.977776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:01.977783 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:01.977823 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:01.981645 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.982530 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.982538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.982542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.982687 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.983103 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.983108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.983112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.983513 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:01.983753 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:01.984326 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 03:16:01.984559 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:16:02.000872 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:01.966671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.14.227.188:42013)
0924 03:16:01.966851 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:01.966857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:01.984946 (+ 18089us) server_negotiation.cc:407] Connection header received
0924 03:16:01.985219 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:01.985226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:01.985302 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:01.985369 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:01.987366 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.988458 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.990358 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:01.991029 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:01.991169 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:01.994391 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:01.994400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:01.994407 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:01.994464 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:01.998019 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.998807 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.998812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.998816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.998976 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:01.999407 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:01.999412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:01.999416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:01.999745 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:01.999962 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:02.000298 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 03:16:02.000506 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:16:02.100010 21391 catalog_manager.cc:4841] TS bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131:38741): tablet 887c0d2676e74194a6cf272b59071bca (table test-table [id=104111a482ed4582903209911bd429b3]) successfully deleted
I20260924 03:16:02.101326 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 21372
I20260924 03:16:02.144898 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:02.126998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.14.227.189:44787)
0924 03:16:02.127251 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:02.127260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:02.128378 (+ 1118us) server_negotiation.cc:407] Connection header received
0924 03:16:02.128659 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:02.128666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:02.128760 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:02.128865 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:02.130529 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:02.131740 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:02.133671 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:02.134335 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:02.134456 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:02.138145 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:02.138152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:02.138159 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:02.138204 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:02.141494 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:02.142249 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:02.142257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:02.142260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:02.142423 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:02.142787 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:02.142793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:02.142799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:02.143073 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:02.143309 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:02.144342 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0924 03:16:02.144548 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:16:02.145203 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:02.128288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.14.227.188:42013)
0924 03:16:02.128565 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:02.128573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:02.130199 (+ 1626us) server_negotiation.cc:407] Connection header received
0924 03:16:02.130460 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:02.130466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:02.130542 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:02.130650 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:02.132499 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:02.133193 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:02.134980 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:02.135596 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:02.135738 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:02.138605 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:02.138612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:02.138617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:02.138657 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:02.142147 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:02.142927 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:02.142933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:02.142936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:02.143082 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:02.143569 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:02.143574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:02.143577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:02.143928 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:02.144201 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:02.144512 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 03:16:02.144724 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 03:16:02.910148 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:02.909086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.131:49619)
0924 03:16:02.909600 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:02.909837 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
W20260924 03:16:02.910564 22654 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:43857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
I20260924 03:16:02.922040 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:02.921190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.130:58631)
0924 03:16:02.921670 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:02.921796 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":310,"threads_started":1}
W20260924 03:16:02.922519 22513 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:43857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
I20260924 03:16:02.924620 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:02.923913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.129:50113)
0924 03:16:02.924354 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:02.924469 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":218,"threads_started":1}
W20260924 03:16:02.924948 22367 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:43857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
I20260924 03:16:03.435221 22726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c892c61c6e3f48c0a38a771ea208748d)
I20260924 03:16:03.435482 22726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } }
I20260924 03:16:03.436750 22726 leader_election.cc:288] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c892c61c6e3f48c0a38a771ea208748d (127.14.227.190:43857), 6fff4654be144b74bb03b1adf4dfca63 (127.14.227.189:44787)
I20260924 03:16:03.438619 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.436930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56082)
0924 03:16:03.437698 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.438373 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":735,"threads_started":1}
W20260924 03:16:03.438930 22173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
I20260924 03:16:03.439524 22730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c892c61c6e3f48c0a38a771ea208748d)
I20260924 03:16:03.439906 22730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } }
I20260924 03:16:03.440620 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.440181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56084)
0924 03:16:03.440374 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.440458 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260924 03:16:03.440796 22173 leader_election.cc:334] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c892c61c6e3f48c0a38a771ea208748d (127.14.227.190:43857): Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
I20260924 03:16:03.441461 22730 leader_election.cc:288] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c892c61c6e3f48c0a38a771ea208748d (127.14.227.190:43857), 163c833a843448b8af4da472b811e5b5 (127.14.227.188:42013)
I20260924 03:16:03.446961 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.437044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.0.0.1:60786)
0924 03:16:03.437978 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.437997 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:03.438200 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:03.438638 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:03.438642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:03.438666 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:03.439002 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.439014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.440592 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.440599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.443396 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.443417 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.445827 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.445836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.446102 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.446154 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 03:16:03.446258 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 03:16:03.446503 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":305,"threads_started":1}
I20260924 03:16:03.447836 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.437107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.14.227.189:44787)
0924 03:16:03.438315 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:03.438322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:03.438341 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:16:03.438431 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:03.438450 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:03.438503 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:03.438609 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:03.439242 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.440438 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.443628 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.445363 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.445596 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.445701 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:03.447175 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0924 03:16:03.447334 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":878,"threads_started":1}
I20260924 03:16:03.448441 22011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "163c833a843448b8af4da472b811e5b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fff4654be144b74bb03b1adf4dfca63" is_pre_election: true
I20260924 03:16:03.448800 22011 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 163c833a843448b8af4da472b811e5b5 in term 1.
I20260924 03:16:03.449455 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.441569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56096)
0924 03:16:03.448898 (+ 7329us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.449180 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7220,"thread_start_us":7456,"threads_started":1}
I20260924 03:16:03.449400 22172 leader_election.cc:302] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 163c833a843448b8af4da472b811e5b5, 6fff4654be144b74bb03b1adf4dfca63; no voters: c892c61c6e3f48c0a38a771ea208748d
I20260924 03:16:03.449819 22726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260924 03:16:03.449869 21987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
I20260924 03:16:03.450006 22726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c892c61c6e3f48c0a38a771ea208748d)
I20260924 03:16:03.450115 22726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:16:03.451503 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.451159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56100)
0924 03:16:03.451308 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.451385 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:16:03.451686 21987 leader_election.cc:334] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c892c61c6e3f48c0a38a771ea208748d (127.14.227.190:43857): Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
I20260924 03:16:03.452925 22726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } }
I20260924 03:16:03.453544 22726 leader_election.cc:288] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [CANDIDATE]: Term 2 election: Requested vote from peers c892c61c6e3f48c0a38a771ea208748d (127.14.227.190:43857), 6fff4654be144b74bb03b1adf4dfca63 (127.14.227.189:44787)
I20260924 03:16:03.454025 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.453620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56102)
0924 03:16:03.453802 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.453880 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 03:16:03.453939 22011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "163c833a843448b8af4da472b811e5b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fff4654be144b74bb03b1adf4dfca63"
I20260924 03:16:03.454155 22011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:16:03.454949 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.454625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56104)
0924 03:16:03.454767 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.454846 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 03:16:03.455152 22173 leader_election.cc:334] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c892c61c6e3f48c0a38a771ea208748d (127.14.227.190:43857): Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
I20260924 03:16:03.455667 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.442055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.14.227.188:42013)
0924 03:16:03.448407 (+ 6352us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:03.448414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:03.449394 (+ 980us) server_negotiation.cc:407] Connection header received
0924 03:16:03.449588 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:03.449594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:03.449679 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:03.449810 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:03.450623 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.451629 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.453167 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.454660 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.454815 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.454899 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:03.455388 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 03:16:03.455475 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6240,"thread_start_us":6387,"threads_started":1}
I20260924 03:16:03.455794 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.441664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:42013 (local address 127.0.0.1:43084)
0924 03:16:03.449296 (+ 7632us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.449310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:03.449450 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:03.450020 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:03.450024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:03.450051 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:03.450379 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.450398 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.451777 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.451784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.453016 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.453027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.455024 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.455041 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.455233 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.455276 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:16:03.455370 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:16:03.455481 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7510,"thread_start_us":6965,"threads_started":1}
I20260924 03:16:03.456342 22198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fff4654be144b74bb03b1adf4dfca63" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "163c833a843448b8af4da472b811e5b5" is_pre_election: true
I20260924 03:16:03.456641 22198 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6fff4654be144b74bb03b1adf4dfca63 in current term 2: Already voted for candidate 163c833a843448b8af4da472b811e5b5 in this term.
I20260924 03:16:03.456923 22011 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 163c833a843448b8af4da472b811e5b5 in term 2.
I20260924 03:16:03.457291 22172 leader_election.cc:302] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 163c833a843448b8af4da472b811e5b5, 6fff4654be144b74bb03b1adf4dfca63; no voters: c892c61c6e3f48c0a38a771ea208748d
I20260924 03:16:03.457270 21986 leader_election.cc:302] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fff4654be144b74bb03b1adf4dfca63; no voters: 163c833a843448b8af4da472b811e5b5, c892c61c6e3f48c0a38a771ea208748d
I20260924 03:16:03.457592 22730 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:16:03.457893 22726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:16:03.458367 22726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [term 2 LEADER]: Becoming Leader. State: Replica: 163c833a843448b8af4da472b811e5b5, State: Running, Role: LEADER
I20260924 03:16:03.458760 22726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } }
I20260924 03:16:03.460729 22736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 163c833a843448b8af4da472b811e5b5. Latest consensus state: current_term: 2 leader_uuid: "163c833a843448b8af4da472b811e5b5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:03.461308 22736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:03.462453 22738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:16:03.466156 22738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:16:03.467015 22738 catalog_manager.cc:1259] Loaded cluster ID: fb31b95fcf71478bbfc2a060611c62e9
I20260924 03:16:03.467115 22738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:16:03.468334 22738 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:16:03.469189 22738 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Loaded TSK: 0
I20260924 03:16:03.470242 22738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:16:03.517254 22188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:16:03.524107 22188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:16:03.528069 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.527696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56136)
0924 03:16:03.527850 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.527952 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 03:16:03.528199 22011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [term 2 FOLLOWER]: Refusing update from remote peer 163c833a843448b8af4da472b811e5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 03:16:03.529021 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.528615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56144)
0924 03:16:03.528820 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.528889 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 03:16:03.529089 22736 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 03:16:03.529374 22173 consensus_peers.cc:487] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 -> Peer c892c61c6e3f48c0a38a771ea208748d (127.14.227.190:43857): Couldn't send request to peer c892c61c6e3f48c0a38a771ea208748d. Status: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 03:16:03.531903 22730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 163c833a843448b8af4da472b811e5b5. Latest consensus state: current_term: 2 leader_uuid: "163c833a843448b8af4da472b811e5b5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:03.532194 22730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:16:03.533468 22736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "163c833a843448b8af4da472b811e5b5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:03.533836 22736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:03.534356 22740 mvcc.cc:205] Tried to move back new op lower bound from 7332740151405072384 to 7332740151130894336. Current Snapshot: MvccSnapshot[applied={T|T < 7332740151405072384}]
I20260924 03:16:03.534924 22730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "163c833a843448b8af4da472b811e5b5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c892c61c6e3f48c0a38a771ea208748d" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43857 } } peers { permanent_uuid: "6fff4654be144b74bb03b1adf4dfca63" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 44787 } attrs { promote: false } } peers { permanent_uuid: "163c833a843448b8af4da472b811e5b5" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 42013 } attrs { promote: false } } }
I20260924 03:16:03.535161 22730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fff4654be144b74bb03b1adf4dfca63 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:16:03.547746 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.541696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.14.227.129:40295)
0924 03:16:03.542132 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:03.542138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:03.542158 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:16:03.542226 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:03.542231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:03.542320 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:03.542987 (+ 667us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:03.543174 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.544067 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.546038 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.546768 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.546962 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.547044 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:03.547320 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 03:16:03.547452 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":405,"threads_started":1}
I20260924 03:16:03.547734 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.541817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:40295 (local address 127.0.0.1:56682)
0924 03:16:03.541973 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.541991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:03.542107 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:03.542484 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:03.542488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:03.542511 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:03.542832 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.542842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.544190 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.544198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.545883 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.545893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.546986 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.546992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.547175 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.547197 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:16:03.547296 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:16:03.547427 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:16:03.548574 22301 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 (DEFAULT_TABLE table=test-table [id=eafd821492f745e29c2b348521919d1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:03.549052 22301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:03.554214 22749 tablet_bootstrap.cc:492] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: Bootstrap starting.
I20260924 03:16:03.555990 22749 tablet_bootstrap.cc:654] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:03.557977 22749 tablet_bootstrap.cc:492] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: No bootstrap required, opened a new log
I20260924 03:16:03.558123 22749 ts_tablet_manager.cc:1397] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260924 03:16:03.558920 22749 raft_consensus.cc:348] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.559193 22749 raft_consensus.cc:374] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:03.559279 22749 raft_consensus.cc:729] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98785d9ec789413a801ce9bdff5ced2c, State: Initialized, Role: FOLLOWER
I20260924 03:16:03.559787 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.542834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:41757 (local address 127.0.0.1:52460)
0924 03:16:03.543427 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.543443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:03.543543 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:03.553345 (+ 9802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:03.553349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:03.553390 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:03.553821 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.553840 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.555078 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.555085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.556723 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.556735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.558638 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.558644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.558799 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.558830 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:16:03.558964 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 03:16:03.559120 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":42,"thread_start_us":600,"threads_started":1}
I20260924 03:16:03.559633 22749 consensus_queue.cc:260] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.559899 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.543346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.14.227.130:41757)
0924 03:16:03.553020 (+ 9674us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:03.553028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:03.553047 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:16:03.553114 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:03.553119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:03.553194 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:03.553308 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:03.554060 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.554942 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.556913 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.558398 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.558546 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.558617 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:03.559185 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 03:16:03.559335 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9530,"thread_start_us":9432,"threads_started":1}
I20260924 03:16:03.560823 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.541332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.14.227.131:38741)
0924 03:16:03.553172 (+ 11840us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:03.553179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:03.553196 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:16:03.553263 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:03.553271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:03.553339 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:03.553473 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:03.554025 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.555226 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.557588 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.558773 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.558967 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.559052 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:03.560341 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0924 03:16:03.560507 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11721,"thread_start_us":11863,"threads_started":1}
I20260924 03:16:03.561237 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.541228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38741 (local address 127.0.0.1:51278)
0924 03:16:03.541413 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.541431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:03.541534 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:03.553495 (+ 11961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:03.553500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:03.553529 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:03.553814 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.553829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.555440 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.555448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.557356 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.557378 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.558947 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.558954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.559189 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.559220 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:16:03.559307 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:16:03.559423 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":48}
I20260924 03:16:03.561014 22447 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 (DEFAULT_TABLE table=test-table [id=eafd821492f745e29c2b348521919d1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:03.562276 22588 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 (DEFAULT_TABLE table=test-table [id=eafd821492f745e29c2b348521919d1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:03.562500 22447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:03.562783 22588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:03.569133 22752 tablet_bootstrap.cc:492] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: Bootstrap starting.
I20260924 03:16:03.570536 22749 ts_tablet_manager.cc:1428] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20260924 03:16:03.572093 22752 tablet_bootstrap.cc:654] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:03.572122 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.571114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.129:32899)
0924 03:16:03.571653 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.571821 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":265,"threads_started":1}
I20260924 03:16:03.576893 22366 heartbeater.cc:503] Master 127.14.227.188:42013 was elected leader, sending a full tablet report...
I20260924 03:16:03.579144 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.572768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.129:40307)
0924 03:16:03.578807 (+ 6039us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.578942 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5895,"threads_started":1}
I20260924 03:16:03.581701 22752 tablet_bootstrap.cc:492] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: No bootstrap required, opened a new log
I20260924 03:16:03.581792 22756 tablet_bootstrap.cc:492] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: Bootstrap starting.
I20260924 03:16:03.581864 22752 ts_tablet_manager.cc:1397] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260924 03:16:03.582631 22752 raft_consensus.cc:348] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.582809 22752 raft_consensus.cc:374] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:03.582942 22752 raft_consensus.cc:729] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96be3ec41d44425a96b119bb6978e3ff, State: Initialized, Role: FOLLOWER
I20260924 03:16:03.583212 22752 consensus_queue.cc:260] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.583500 22756 tablet_bootstrap.cc:654] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:03.586053 22752 ts_tablet_manager.cc:1428] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:16:03.586546 22512 heartbeater.cc:503] Master 127.14.227.188:42013 was elected leader, sending a full tablet report...
I20260924 03:16:03.587595 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.584248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.130:36651)
0924 03:16:03.587158 (+ 2910us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.587351 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2780,"thread_start_us":2645,"threads_started":1}
I20260924 03:16:03.588958 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.588579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.130:39847)
0924 03:16:03.588726 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.588807 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:16:03.591151 22756 tablet_bootstrap.cc:492] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: No bootstrap required, opened a new log
I20260924 03:16:03.591305 22756 ts_tablet_manager.cc:1397] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 03:16:03.591913 22756 raft_consensus.cc:348] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.592133 22756 raft_consensus.cc:374] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:03.592242 22756 raft_consensus.cc:729] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb775656b7d9489fa30ee0eeab6484bb, State: Initialized, Role: FOLLOWER
I20260924 03:16:03.592478 22756 consensus_queue.cc:260] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.593207 22756 ts_tablet_manager.cc:1428] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:16:03.594370 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.593489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.131:55553)
0924 03:16:03.594041 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.594187 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
I20260924 03:16:03.595170 22653 heartbeater.cc:503] Master 127.14.227.188:42013 was elected leader, sending a full tablet report...
I20260924 03:16:03.595376 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.594923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.131:57289)
0924 03:16:03.595086 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.595143 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 03:16:03.900542 22751 raft_consensus.cc:482] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:16:03.900743 22751 raft_consensus.cc:504] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.902240 22751 leader_election.cc:288] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131:38741), 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757)
I20260924 03:16:03.910230 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.903357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:51687 (local address 127.14.227.130:41757)
0924 03:16:03.903614 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:03.903620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:03.903638 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:16:03.903707 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:03.903714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:03.903787 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:03.903859 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:03.904445 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.905451 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.907977 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.909196 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.909349 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.909419 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:03.909790 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 03:16:03.909968 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:16:03.910286 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.902805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38741 (local address 127.14.227.129:45347)
0924 03:16:03.903016 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.903038 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:03.903183 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:03.903601 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:03.903607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:03.903633 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:03.904017 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.904032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.905360 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.905363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.907808 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.907826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.909394 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.909400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.909598 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.909639 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:16:03.909735 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:16:03.909880 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":161}
I20260924 03:16:03.910205 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.902777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:45347 (local address 127.14.227.131:38741)
0924 03:16:03.902974 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:03.902979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:03.903135 (+ 156us) server_negotiation.cc:407] Connection header received
0924 03:16:03.903374 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:03.903378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:03.903475 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:03.903571 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:03.904195 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.905239 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.907977 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.909190 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.909344 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.909419 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:03.909790 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 03:16:03.909930 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:16:03.910923 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.903207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:41757 (local address 127.14.227.129:51687)
0924 03:16:03.903379 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.903398 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:03.903523 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:03.903912 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:03.903918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:03.903945 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:03.904305 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.904313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.905571 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.905577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.907820 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:03.907828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:03.909472 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:03.909479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:03.909650 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:03.909678 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 03:16:03.909753 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:16:03.909887 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":135}
I20260924 03:16:03.911247 22608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2842c3cea14c02b9f7b501eb6a2fe9" candidate_uuid: "98785d9ec789413a801ce9bdff5ced2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb775656b7d9489fa30ee0eeab6484bb" is_pre_election: true
I20260924 03:16:03.911767 22608 raft_consensus.cc:2454] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98785d9ec789413a801ce9bdff5ced2c in term 0.
I20260924 03:16:03.911760 22467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2842c3cea14c02b9f7b501eb6a2fe9" candidate_uuid: "98785d9ec789413a801ce9bdff5ced2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be3ec41d44425a96b119bb6978e3ff" is_pre_election: true
I20260924 03:16:03.912000 22467 raft_consensus.cc:2454] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98785d9ec789413a801ce9bdff5ced2c in term 0.
I20260924 03:16:03.912446 22257 leader_election.cc:302] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98785d9ec789413a801ce9bdff5ced2c, bb775656b7d9489fa30ee0eeab6484bb; no voters:
I20260924 03:16:03.912796 22751 raft_consensus.cc:2781] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:16:03.912892 22751 raft_consensus.cc:482] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:16:03.913065 22751 raft_consensus.cc:3037] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:03.915441 22751 raft_consensus.cc:504] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.915957 22751 leader_election.cc:288] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [CANDIDATE]: Term 1 election: Requested vote from peers bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131:38741), 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757)
I20260924 03:16:03.916373 22608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2842c3cea14c02b9f7b501eb6a2fe9" candidate_uuid: "98785d9ec789413a801ce9bdff5ced2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb775656b7d9489fa30ee0eeab6484bb"
I20260924 03:16:03.916482 22467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2842c3cea14c02b9f7b501eb6a2fe9" candidate_uuid: "98785d9ec789413a801ce9bdff5ced2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96be3ec41d44425a96b119bb6978e3ff"
I20260924 03:16:03.916569 22608 raft_consensus.cc:3037] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:03.916706 22467 raft_consensus.cc:3037] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:03.918684 22608 raft_consensus.cc:2454] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98785d9ec789413a801ce9bdff5ced2c in term 1.
I20260924 03:16:03.918725 22467 raft_consensus.cc:2454] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98785d9ec789413a801ce9bdff5ced2c in term 1.
I20260924 03:16:03.919061 22257 leader_election.cc:302] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98785d9ec789413a801ce9bdff5ced2c, bb775656b7d9489fa30ee0eeab6484bb; no voters:
I20260924 03:16:03.919337 22751 raft_consensus.cc:2781] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:03.919797 22751 raft_consensus.cc:686] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 1 LEADER]: Becoming Leader. State: Replica: 98785d9ec789413a801ce9bdff5ced2c, State: Running, Role: LEADER
I20260924 03:16:03.920403 22751 consensus_queue.cc:237] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } }
I20260924 03:16:03.920713 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:03.920351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.129:59019)
0924 03:16:03.920482 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:03.920581 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:16:03.923764 22188 catalog_manager.cc:5495] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129). New cstate: current_term: 1 leader_uuid: "98785d9ec789413a801ce9bdff5ced2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98785d9ec789413a801ce9bdff5ced2c" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 } health_report { overall_health: UNKNOWN } } }
I20260924 03:16:04.012392 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.007023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.14.227.129:40295)
0924 03:16:04.007250 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:04.007255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:04.007274 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:16:04.007444 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:04.007449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:04.007527 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:04.007681 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:04.008044 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:04.008745 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:04.010099 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:04.010619 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:04.010746 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:04.011158 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:04.011785 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:04.011932 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:04.012037 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 03:16:04.012115 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:16:04.017033 22467 raft_consensus.cc:1261] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 1 FOLLOWER]: Refusing update from remote peer 98785d9ec789413a801ce9bdff5ced2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:04.017397 22608 raft_consensus.cc:1261] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 1 FOLLOWER]: Refusing update from remote peer 98785d9ec789413a801ce9bdff5ced2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:04.018323 22762 consensus_queue.cc:1035] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96be3ec41d44425a96b119bb6978e3ff" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:04.019090 22751 consensus_queue.cc:1035] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb775656b7d9489fa30ee0eeab6484bb" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:04.019305 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.018907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56148)
0924 03:16:04.019052 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.019129 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:16:04.020488 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.020088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56152)
0924 03:16:04.020273 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.020367 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 03:16:04.021703 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.021077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.130:43297)
0924 03:16:04.021335 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.021477 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260924 03:16:04.023435 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.023004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.131:52695)
0924 03:16:04.023176 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.023272 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 03:16:04.026197 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.025716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.129:44207)
0924 03:16:04.025936 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.026044 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 03:16:04.028734 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.028200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.130:55045)
0924 03:16:04.028444 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.028543 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 03:16:04.029572 22765 mvcc.cc:205] Tried to move back new op lower bound from 7332740153406410752 to 7332740153022128128. Current Snapshot: MvccSnapshot[applied={T|T < 7332740153406410752 or (T in {7332740153406410752})}]
I20260924 03:16:04.037483 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.036919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.129:52475)
0924 03:16:04.037172 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.037275 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 03:16:04.039741 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.039372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.14.227.131:48057)
0924 03:16:04.039506 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.039600 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:16:04.049425 22188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:04.049731 22188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:04.055389 22188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5: Sending DeleteTablet for 3 replicas of tablet 0c2842c3cea14c02b9f7b501eb6a2fe9
I20260924 03:16:04.056293 22301 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:04 UTC) from {username='oryx', principal='oryx/127.14.227.188@KRBTEST.COM'} at 127.0.0.1:56682
I20260924 03:16:04.056316 22588 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:04 UTC) from {username='oryx', principal='oryx/127.14.227.188@KRBTEST.COM'} at 127.0.0.1:51278
I20260924 03:16:04.056684 22447 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:04 UTC) from {username='oryx', principal='oryx/127.14.227.188@KRBTEST.COM'} at 127.0.0.1:52460
I20260924 03:16:04.058578 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 22175
I20260924 03:16:04.070376 22775 tablet_replica.cc:330] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: stopping tablet replica
I20260924 03:16:04.070613 22775 raft_consensus.cc:2229] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:16:04.071224 22775 raft_consensus.cc:2258] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:04.071861 22776 tablet_replica.cc:330] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: stopping tablet replica
I20260924 03:16:04.072075 22776 raft_consensus.cc:2229] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:04.072360 22776 raft_consensus.cc:2258] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:04.073385 22776 ts_tablet_manager.cc:1905] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:04.074519 22775 ts_tablet_manager.cc:1905] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:04.079663 22776 ts_tablet_manager.cc:1918] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:04.079789 22776 log.cc:1196] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/0c2842c3cea14c02b9f7b501eb6a2fe9
I20260924 03:16:04.080243 22776 ts_tablet_manager.cc:1939] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P bb775656b7d9489fa30ee0eeab6484bb: Deleting consensus metadata
I20260924 03:16:04.081424 22174 catalog_manager.cc:4841] TS bb775656b7d9489fa30ee0eeab6484bb (127.14.227.131:38741): tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 (table test-table [id=eafd821492f745e29c2b348521919d1e]) successfully deleted
I20260924 03:16:04.081693 22775 ts_tablet_manager.cc:1918] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:04.081866 22775 log.cc:1196] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/0c2842c3cea14c02b9f7b501eb6a2fe9
I20260924 03:16:04.082356 22775 ts_tablet_manager.cc:1939] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 98785d9ec789413a801ce9bdff5ced2c: Deleting consensus metadata
I20260924 03:16:04.083364 22171 catalog_manager.cc:4841] TS 98785d9ec789413a801ce9bdff5ced2c (127.14.227.129:40295): tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 (table test-table [id=eafd821492f745e29c2b348521919d1e]) successfully deleted
I20260924 03:16:04.084225 22777 tablet_replica.cc:330] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: stopping tablet replica
I20260924 03:16:04.084430 22777 raft_consensus.cc:2229] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:04.084757 22777 raft_consensus.cc:2258] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:04.085700 22777 ts_tablet_manager.cc:1905] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:04.092175 22777 ts_tablet_manager.cc:1918] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:04.092308 22777 log.cc:1196] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/0c2842c3cea14c02b9f7b501eb6a2fe9
I20260924 03:16:04.092758 22777 ts_tablet_manager.cc:1939] T 0c2842c3cea14c02b9f7b501eb6a2fe9 P 96be3ec41d44425a96b119bb6978e3ff: Deleting consensus metadata
I20260924 03:16:04.093868 22174 catalog_manager.cc:4841] TS 96be3ec41d44425a96b119bb6978e3ff (127.14.227.130:41757): tablet 0c2842c3cea14c02b9f7b501eb6a2fe9 (table test-table [id=eafd821492f745e29c2b348521919d1e]) successfully deleted
I20260924 03:16:04.096906 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.059964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.14.227.129:40295)
0924 03:16:04.060158 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:04.060167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:04.083401 (+ 23234us) server_negotiation.cc:407] Connection header received
0924 03:16:04.083893 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:04.083896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:04.083978 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:04.084094 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:04.085743 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:04.086621 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:04.087789 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:04.088193 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:04.088346 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:04.090869 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:04.090885 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:04.090909 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:04.090951 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:04.094255 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:04.095082 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:04.095089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:04.095091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:04.095243 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:04.095638 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:04.095640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:04.095642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:04.095946 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:04.096162 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:04.096478 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 03:16:04.096627 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:16:04.097503 22361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:04.196475 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 22175
I20260924 03:16:04.214461 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 22384
I20260924 03:16:04.227996 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.215329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.14.227.130:41757)
0924 03:16:04.215501 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:04.215506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:04.215605 (+ 99us) server_negotiation.cc:407] Connection header received
0924 03:16:04.215754 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:04.215760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:04.215851 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:04.215953 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:04.217309 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:04.218565 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:04.219748 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:04.220341 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:04.220456 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:04.223080 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:04.223096 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:04.223112 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:04.223145 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:04.225720 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:04.226397 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:04.226403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:04.226405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:04.226529 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:04.226828 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:04.226830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:04.226832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:04.227142 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:04.227345 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:04.227566 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 03:16:04.227709 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:16:04.228643 22507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:04.319211 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 22384
I20260924 03:16:04.335351 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 22525
I20260924 03:16:04.349711 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.336084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.14.227.131:38741)
0924 03:16:04.336267 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:04.336274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:04.336348 (+ 74us) server_negotiation.cc:407] Connection header received
0924 03:16:04.336544 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:04.336556 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:04.336626 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:04.336745 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:04.338352 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:04.339275 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:04.340790 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:04.341310 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:04.341445 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:04.344214 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:04.344232 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:04.344259 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:04.344293 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:04.346932 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:04.347622 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:04.347631 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:04.347635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:04.347789 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:04.348170 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:04.348177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:04.348183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:04.348564 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:04.348888 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:04.349166 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 03:16:04.349377 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:16:04.350425 22648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:04.468614 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 22525
I20260924 03:16:04.490203 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 21865
I20260924 03:16:04.491282 22031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:04.522322 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.521569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56166)
0924 03:16:04.522043 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.522129 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":198,"threads_started":1}
I20260924 03:16:04.526983 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.526629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43857 (local address 127.0.0.1:56168)
0924 03:16:04.526793 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.526867 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 03:16:04.607415 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 21865
W20260924 03:16:04.628652 22172 connection.cc:535] client connection to 127.14.227.189:44787 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:16:04.628953 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 21965
I20260924 03:16:04.629863 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:04.629449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:44787 (local address 127.0.0.1:60798)
0924 03:16:04.629625 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:04.629710 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:44787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 03:16:04.630061 22172 consensus_peers.cc:487] T 00000000000000000000000000000000 P 163c833a843448b8af4da472b811e5b5 -> Peer 6fff4654be144b74bb03b1adf4dfca63 (127.14.227.189:44787): Couldn't send request to peer 6fff4654be144b74bb03b1adf4dfca63. Status: Network error: Client connection negotiation failed: client connection to 127.14.227.189:44787: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 03:16:04.630069 22218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:04.749328 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 21965
2026-09-24T03:16:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9630 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:16:04 dist-test-slave-vd0p krb5kdc[22788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:16:04 dist-test-slave-vd0p krb5kdc[22788](info): set up 2 sockets
Sep 24 03:16:04 dist-test-slave-vd0p krb5kdc[22788](info): commencing operation
krb5kdc: starting...
W20260924 03:16:05.854962 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:16:05 dist-test-slave-vd0p krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:16:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.190.keytab.
Entry for principal oryx/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.190.keytab.
I20260924 03:16:06.056886 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38633
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.190.keytab
--principal=oryx/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:06.363588 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:06.363901 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:06.364085 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:06.371273 22801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:16:06.371405 22801 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 03:16:06.371445 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:06.371522 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:06.371592 22801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:16:06.371709 22801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:16:06.381544 22801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:38633
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.190.keytab
--principal=oryx/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:06.382879 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:06.384142 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:06.396008 22807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:06.396041 22806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:06.395953 22809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:06.396713 22801 server_base.cc:1034] running on GCE node
I20260924 03:16:06.397418 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:06.398770 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:06.399981 22801 hybrid_clock.cc:648] HybridClock initialized: now 1790219766399922 us; error 27 us; skew 500 ppm
I20260924 03:16:06.400326 22801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:06 dist-test-slave-vd0p krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219766, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:06.403276 22801 init.cc:377] Logged in from keytab as oryx/127.14.227.190@KRBTEST.COM (short username oryx)
I20260924 03:16:06.404769 22801 webserver.cc:460] Webserver started at http://127.14.227.190:37325/ using document root <none> and password file <none>
I20260924 03:16:06.405472 22801 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:06.405601 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:06.405938 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:06.408727 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "fbe754dc06da4684b4935e59ff045c23"
format_stamp: "Formatted at 2026-09-24 03:16:06 on dist-test-slave-vd0p"
server_key: "f27bcf8414e0edb55a661bc9d9ddcf0f"
server_key_iv: "862fd8f8e908abc1740a31091df8862f"
server_key_version: "encryptionkey@0"
I20260924 03:16:06.409487 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "fbe754dc06da4684b4935e59ff045c23"
format_stamp: "Formatted at 2026-09-24 03:16:06 on dist-test-slave-vd0p"
server_key: "f27bcf8414e0edb55a661bc9d9ddcf0f"
server_key_iv: "862fd8f8e908abc1740a31091df8862f"
server_key_version: "encryptionkey@0"
I20260924 03:16:06.414448 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 03:16:06.418095 22816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:06.419286 22801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:16:06.419459 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "fbe754dc06da4684b4935e59ff045c23"
format_stamp: "Formatted at 2026-09-24 03:16:06 on dist-test-slave-vd0p"
server_key: "f27bcf8414e0edb55a661bc9d9ddcf0f"
server_key_iv: "862fd8f8e908abc1740a31091df8862f"
server_key_version: "encryptionkey@0"
I20260924 03:16:06.419663 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:06.435474 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:06.439149 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:06.439416 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:06.460271 22801 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:38633
I20260924 03:16:06.460278 22877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:38633 every 8 connection(s)
I20260924 03:16:06.461651 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:16:06.465281 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:06.465880 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 22801
I20260924 03:16:06.466101 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:16:06.466940 15246 external_mini_cluster.cc:1428] Setting key d851e5ae3ecac79f704c31e3f3f7e525
I20260924 03:16:06.479187 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: Bootstrap starting.
Sep 24 03:16:06 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:16:06.482712 22878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:06.483920 22878 log.cc:824] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:06.486652 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: No bootstrap required, opened a new log
I20260924 03:16:06.489285 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:06.468980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.14.227.190:38633)
0924 03:16:06.469774 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:06.469789 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:06.469830 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:16:06.471583 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:06.471605 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:06.471992 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:06.472305 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:06.473942 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:06.475265 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:06.477044 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:06.477681 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:06.477808 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:06.482373 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:06.482381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:06.482395 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:06.482428 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:06.485425 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:06.486085 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:06.486091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:06.486097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:06.486267 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:06.486643 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:06.486648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:06.486653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:06.487190 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:06.487486 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:06.487891 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 03:16:06.488308 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":597,"thread_start_us":383,"threads_started":1}
I20260924 03:16:06.494239 22878 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbe754dc06da4684b4935e59ff045c23" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38633 } }
I20260924 03:16:06.494582 22878 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:06.494699 22878 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe754dc06da4684b4935e59ff045c23, State: Initialized, Role: FOLLOWER
I20260924 03:16:06.495260 22878 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbe754dc06da4684b4935e59ff045c23" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38633 } }
I20260924 03:16:06.495512 22878 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:16:06.495637 22878 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:16:06.495795 22878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:06.498426 22878 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbe754dc06da4684b4935e59ff045c23" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38633 } }
I20260924 03:16:06.498911 22878 leader_election.cc:302] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbe754dc06da4684b4935e59ff045c23; no voters:
I20260924 03:16:06.499436 22878 leader_election.cc:288] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:16:06.499604 22883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:06.500309 22883 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [term 1 LEADER]: Becoming Leader. State: Replica: fbe754dc06da4684b4935e59ff045c23, State: Running, Role: LEADER
I20260924 03:16:06.500783 22883 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbe754dc06da4684b4935e59ff045c23" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38633 } }
I20260924 03:16:06.501415 22878 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:16:06.503584 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbe754dc06da4684b4935e59ff045c23. Latest consensus state: current_term: 1 leader_uuid: "fbe754dc06da4684b4935e59ff045c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbe754dc06da4684b4935e59ff045c23" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38633 } } }
I20260924 03:16:06.504052 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:06.504118 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbe754dc06da4684b4935e59ff045c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbe754dc06da4684b4935e59ff045c23" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38633 } } }
I20260924 03:16:06.504407 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:06.506028 22890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:16:06.510883 22890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:16:06.520153 22890 catalog_manager.cc:1347] Generated new cluster ID: cc2d80df4c4a43748f912a71e448ef44
I20260924 03:16:06.520259 22890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:16:06.533900 22890 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:16:06.535332 22890 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:16:06.549315 22890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: Generated new TSK 0
I20260924 03:16:06.550261 22890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab.
Entry for principal oryx/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab.
I20260924 03:16:06.609841 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38633
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab
--principal=oryx/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:06.925773 22904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:06.926155 22904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:06.926335 22904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:06.933135 22904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:06.933260 22904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:06.933408 22904 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:16:06.933545 22904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:16:06.942946 22904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab
--principal=oryx/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:06.944150 22904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:06.945452 22904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:06.958596 22909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:06.958635 22910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:06.958668 22912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:06.959492 22904 server_base.cc:1034] running on GCE node
I20260924 03:16:06.960067 22904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:06.960978 22904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:06.962203 22904 hybrid_clock.cc:648] HybridClock initialized: now 1790219766962117 us; error 64 us; skew 500 ppm
I20260924 03:16:06.962479 22904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:06 dist-test-slave-vd0p krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219766, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:06.965438 22904 init.cc:377] Logged in from keytab as oryx/127.14.227.129@KRBTEST.COM (short username oryx)
I20260924 03:16:06.966948 22904 webserver.cc:460] Webserver started at http://127.14.227.129:45685/ using document root <none> and password file <none>
I20260924 03:16:06.967661 22904 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:06.967794 22904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:06.968111 22904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:06.971061 22904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "64f5bf792d454d54b85ec47d4715dc54"
format_stamp: "Formatted at 2026-09-24 03:16:06 on dist-test-slave-vd0p"
server_key: "0d42368f18a765a4e8c2d0ea970050d2"
server_key_iv: "7fad126f6822ca9303dec2968d50422b"
server_key_version: "encryptionkey@0"
I20260924 03:16:06.971832 22904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "64f5bf792d454d54b85ec47d4715dc54"
format_stamp: "Formatted at 2026-09-24 03:16:06 on dist-test-slave-vd0p"
server_key: "0d42368f18a765a4e8c2d0ea970050d2"
server_key_iv: "7fad126f6822ca9303dec2968d50422b"
server_key_version: "encryptionkey@0"
I20260924 03:16:06.976614 22904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:16:06.979882 22919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:06.981197 22904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:16:06.981364 22904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "64f5bf792d454d54b85ec47d4715dc54"
format_stamp: "Formatted at 2026-09-24 03:16:06 on dist-test-slave-vd0p"
server_key: "0d42368f18a765a4e8c2d0ea970050d2"
server_key_iv: "7fad126f6822ca9303dec2968d50422b"
server_key_version: "encryptionkey@0"
I20260924 03:16:06.981585 22904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:06.999615 22904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:07.003854 22904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:07.004134 22904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:07.007122 22904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:07.007241 22904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:07.007380 22904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:07.007488 22904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 03:16:07 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219766, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:16:07.034583 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:07.007681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:37417 (local address 127.14.227.190:38633)
0924 03:16:07.009307 (+ 1626us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:07.009317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:07.010513 (+ 1196us) server_negotiation.cc:407] Connection header received
0924 03:16:07.012652 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:07.012659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:07.012760 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:07.012903 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:07.015251 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.016506 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.019828 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.020440 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.020548 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:07.026341 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:07.026351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:07.026358 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:07.026407 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:07.030293 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:07.031424 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:07.031432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:07.031437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:07.031597 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:07.032276 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:07.032283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:07.032288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:07.032556 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:07.032663 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:07.033848 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0924 03:16:07.034127 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451,"thread_start_us":1314,"threads_started":1}
I20260924 03:16:07.036341 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:07.008380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38633 (local address 127.14.227.129:37417)
0924 03:16:07.010011 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:07.010060 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:07.012285 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:07.013163 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:07.013177 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:07.014062 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:07.015019 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:07.015053 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.016695 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.016704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:07.019658 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:07.019678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.020744 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.020754 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:07.021139 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:07.022049 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:07.022083 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:07.026071 (+ 3988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:07.030473 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:07.030486 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:07.030505 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:07.031266 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:07.031726 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:07.031734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:07.031740 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:07.032130 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:07.032688 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:07.032703 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:07.033285 (+ 582us) client_negotiation.cc:769] Sending connection context
0924 03:16:07.033680 (+ 395us) client_negotiation.cc:240] Negotiation successful
0924 03:16:07.034420 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1280,"thread_start_us":976,"threads_started":1}
I20260924 03:16:07.051327 22904 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:40689
I20260924 03:16:07.051349 23038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:40689 every 8 connection(s)
I20260924 03:16:07.052927 22904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:16:07.055258 23040 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38633
I20260924 03:16:07.055617 23040 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:07.056478 23040 heartbeater.cc:511] Master 127.14.227.190:38633 requested a full tablet report, sending...
I20260924 03:16:07.058454 22833 ts_manager.cc:194] Registered new tserver with Master: 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689)
I20260924 03:16:07.060241 22833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:37417
I20260924 03:16:07.062438 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 22904
I20260924 03:16:07.062651 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:16:07.063406 15246 external_mini_cluster.cc:1428] Setting key 27681ca5328d4f8ec2e8fac0bd2a7af8
I20260924 03:16:07.066165 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:07.052524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:57625 (local address 127.14.227.190:38633)
0924 03:16:07.052725 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:07.052732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:07.054468 (+ 1736us) server_negotiation.cc:407] Connection header received
0924 03:16:07.054745 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:07.054751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:07.054854 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:07.054956 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:07.056364 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.057128 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.058562 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.059147 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.059265 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:07.060879 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:07.060885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:07.060889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:07.060934 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:07.063574 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:07.064302 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:07.064310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:07.064313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:07.064454 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:07.064805 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:07.064810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:07.064814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:07.065172 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:07.065317 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:07.065710 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 03:16:07.065866 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:16:07.066395 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:07.052556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38633 (local address 127.14.227.129:57625)
0924 03:16:07.054336 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:07.054361 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:07.054599 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:07.055145 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:07.055151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:07.055818 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:07.056213 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:07.056226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.057312 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.057318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:07.058416 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:07.058440 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.059299 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.059303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:07.059411 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:07.059987 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:07.060003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:07.060700 (+ 697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:07.063697 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:07.063701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:07.063705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:07.064195 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:07.064585 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:07.064588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:07.064591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:07.064703 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:07.065294 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:07.065297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:07.065480 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 03:16:07.065631 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 03:16:07.065904 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"thread_start_us":1364,"threads_started":1}
WARNING: no policy specified for oryx/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab.
Entry for principal oryx/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab.
I20260924 03:16:07.107220 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38633
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab
--principal=oryx/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:07.436357 23044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:07.436604 23044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:07.436702 23044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:07.443933 23044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:07.444028 23044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:07.444108 23044 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:16:07.444185 23044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:16:07.453454 23044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab
--principal=oryx/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:07.454588 23044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:07.455835 23044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:07.468793 23050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:07.468927 23052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:07.469342 23049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:07.469534 23044 server_base.cc:1034] running on GCE node
I20260924 03:16:07.470256 23044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:07.471160 23044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:07.472400 23044 hybrid_clock.cc:648] HybridClock initialized: now 1790219767472330 us; error 47 us; skew 500 ppm
I20260924 03:16:07.472703 23044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:07 dist-test-slave-vd0p krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219767, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:07.475872 23044 init.cc:377] Logged in from keytab as oryx/127.14.227.130@KRBTEST.COM (short username oryx)
I20260924 03:16:07.477350 23044 webserver.cc:460] Webserver started at http://127.14.227.130:35377/ using document root <none> and password file <none>
I20260924 03:16:07.478098 23044 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:07.478233 23044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:07.478549 23044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:07.481349 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd"
format_stamp: "Formatted at 2026-09-24 03:16:07 on dist-test-slave-vd0p"
server_key: "50a015216f3de2b2c1c38319887398e0"
server_key_iv: "ac92df07d70f75f7c441f321def28810"
server_key_version: "encryptionkey@0"
I20260924 03:16:07.482170 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd"
format_stamp: "Formatted at 2026-09-24 03:16:07 on dist-test-slave-vd0p"
server_key: "50a015216f3de2b2c1c38319887398e0"
server_key_iv: "ac92df07d70f75f7c441f321def28810"
server_key_version: "encryptionkey@0"
I20260924 03:16:07.487247 23044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:16:07.490485 23059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:07.491714 23044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:16:07.491881 23044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd"
format_stamp: "Formatted at 2026-09-24 03:16:07 on dist-test-slave-vd0p"
server_key: "50a015216f3de2b2c1c38319887398e0"
server_key_iv: "ac92df07d70f75f7c441f321def28810"
server_key_version: "encryptionkey@0"
I20260924 03:16:07.492089 23044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:07.516862 23044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:07.520546 23044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:07.520810 23044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:07.523571 23044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:07.523702 23044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:07.523838 23044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:07.523924 23044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:16:07 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219767, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:16:07.546296 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:07.523788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:51417 (local address 127.14.227.190:38633)
0924 03:16:07.523989 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:07.523996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:07.525496 (+ 1500us) server_negotiation.cc:407] Connection header received
0924 03:16:07.527730 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:07.527740 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:07.527825 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:07.527942 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:07.529904 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.531197 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.533632 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.534166 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.534320 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:07.539303 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:07.539310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:07.539315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:07.539355 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:07.542418 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:07.543512 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:07.543516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:07.543518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:07.543631 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:07.544232 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:07.544237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:07.544240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:07.544569 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:07.544731 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:07.545713 (+ 982us) server_negotiation.cc:299] Negotiation successful
0924 03:16:07.545935 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:16:07.547204 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:07.524469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38633 (local address 127.14.227.130:51417)
0924 03:16:07.525314 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:07.525365 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:07.527402 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:07.528163 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:07.528171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:07.528869 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:07.529689 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:07.529719 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.531356 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.531363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:07.533483 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:07.533497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.534463 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.534474 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:07.534828 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:07.535616 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:07.535653 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:07.539099 (+ 3446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:07.542571 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:07.542583 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:07.542605 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:07.543377 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:07.543775 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:07.543784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:07.543794 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:07.544102 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:07.544832 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:07.544856 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:07.545266 (+ 410us) client_negotiation.cc:769] Sending connection context
0924 03:16:07.545624 (+ 358us) client_negotiation.cc:240] Negotiation successful
0924 03:16:07.546152 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":375,"threads_started":1}
I20260924 03:16:07.568145 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:07.554049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:37959 (local address 127.14.227.190:38633)
0924 03:16:07.554215 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:07.554222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:07.554901 (+ 679us) server_negotiation.cc:407] Connection header received
0924 03:16:07.555012 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:07.555020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:07.555142 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:07.555266 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:07.556162 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.557564 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.559319 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.559911 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.560054 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:07.561723 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:07.561729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:07.561734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:07.561771 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:07.565102 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:07.566028 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:07.566032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:07.566034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:07.566168 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:07.566614 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:07.566619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:07.566622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:07.566957 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:07.567106 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:07.567549 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 03:16:07.567771 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:16:07.568264 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:07.554161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38633 (local address 127.14.227.130:37959)
0924 03:16:07.554800 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:07.554816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:07.554919 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:07.555274 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:07.555277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:07.555735 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:07.556027 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:07.556035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.557705 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.557708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:07.559130 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:07.559149 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:07.560099 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:07.560105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:07.560260 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:07.560856 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:07.560871 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:07.561552 (+ 681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:07.565242 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:07.565250 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:07.565257 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:07.565871 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:07.566280 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:07.566284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:07.566287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:07.566453 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:07.567079 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:07.567084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:07.567262 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 03:16:07.567434 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 03:16:07.567815 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":354,"threads_started":1}
I20260924 03:16:07.579277 23044 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:32981
I20260924 03:16:07.579285 23178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:32981 every 8 connection(s)
I20260924 03:16:07.580447 23044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:16:07.582456 23179 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38633
I20260924 03:16:07.582625 23179 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:07.583145 23179 heartbeater.cc:511] Master 127.14.227.190:38633 requested a full tablet report, sending...
I20260924 03:16:07.584275 22833 ts_manager.cc:194] Registered new tserver with Master: cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:07.585135 22833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:51417
I20260924 03:16:07.586292 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 23044
I20260924 03:16:07.586472 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:16:07.586907 15246 external_mini_cluster.cc:1428] Setting key 7a8a3f0b4517c898ebe9a933a259b2ca
WARNING: no policy specified for oryx/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab.
Entry for principal oryx/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab.
I20260924 03:16:07.629757 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38633
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab
--principal=oryx/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:07.938201 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:07.938444 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:07.938613 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:07.945673 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:07.945773 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:07.945919 23183 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 03:16:07.946095 23183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:16:07.955806 23183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab
--principal=oryx/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.227.190:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:07.956964 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:07.958287 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:07.970899 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:07.970989 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:07.970966 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:07.971484 23183 server_base.cc:1034] running on GCE node
I20260924 03:16:07.972229 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:07.973067 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:07.974291 23183 hybrid_clock.cc:648] HybridClock initialized: now 1790219767974232 us; error 56 us; skew 500 ppm
I20260924 03:16:07.974567 23183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:07 dist-test-slave-vd0p krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219767, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:07.977398 23183 init.cc:377] Logged in from keytab as oryx/127.14.227.131@KRBTEST.COM (short username oryx)
I20260924 03:16:07.978888 23183 webserver.cc:460] Webserver started at http://127.14.227.131:44195/ using document root <none> and password file <none>
I20260924 03:16:07.979527 23183 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:07.979655 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:07.979956 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:07.982744 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "3943d37a1703465093f902bd7289e278"
format_stamp: "Formatted at 2026-09-24 03:16:07 on dist-test-slave-vd0p"
server_key: "cf589e97673cf82a0d9cc051cc44a77e"
server_key_iv: "b018aac912822c595062aef7c10c0027"
server_key_version: "encryptionkey@0"
I20260924 03:16:07.983512 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "3943d37a1703465093f902bd7289e278"
format_stamp: "Formatted at 2026-09-24 03:16:07 on dist-test-slave-vd0p"
server_key: "cf589e97673cf82a0d9cc051cc44a77e"
server_key_iv: "b018aac912822c595062aef7c10c0027"
server_key_version: "encryptionkey@0"
I20260924 03:16:07.988508 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:16:07.991813 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:07.993036 23183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:16:07.993194 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "3943d37a1703465093f902bd7289e278"
format_stamp: "Formatted at 2026-09-24 03:16:07 on dist-test-slave-vd0p"
server_key: "cf589e97673cf82a0d9cc051cc44a77e"
server_key_iv: "b018aac912822c595062aef7c10c0027"
server_key_version: "encryptionkey@0"
I20260924 03:16:07.993408 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:08.016609 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:08.020267 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:08.020481 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:08.022976 23183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:08.023051 23183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:08.023231 23183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:08.023288 23183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 03:16:08 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219767, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:16:08.046576 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.023554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:41053 (local address 127.14.227.190:38633)
0924 03:16:08.023759 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.023766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.025242 (+ 1476us) server_negotiation.cc:407] Connection header received
0924 03:16:08.027551 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.027558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.027634 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.027691 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:08.030051 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.031117 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.033469 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.034102 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.034237 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.039244 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:08.039251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:08.039256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:08.039341 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:08.042746 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:08.043734 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:08.043742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:08.043748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:08.043875 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:08.044458 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:08.044462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:08.044466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:08.044826 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:08.044980 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.046000 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.046212 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:16:08.050630 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.024172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38633 (local address 127.14.227.131:41053)
0924 03:16:08.025066 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.025108 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.027179 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.028207 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.028214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.028957 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:08.029836 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.029877 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.031288 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.031293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.033334 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.033345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.034351 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.034359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.034572 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.035437 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:08.035481 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:08.039000 (+ 3519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:08.042925 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:08.042933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:08.042950 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:08.043604 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:08.043986 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:08.043990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:08.043997 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:08.044301 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:08.045055 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:08.045062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:08.045572 (+ 510us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.045858 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.046410 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":386,"threads_started":1}
I20260924 03:16:08.063870 23040 heartbeater.cc:503] Master 127.14.227.190:38633 was elected leader, sending a full tablet report...
I20260924 03:16:08.067431 23183 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:32783
I20260924 03:16:08.067528 23317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:32783 every 8 connection(s)
I20260924 03:16:08.068759 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:16:08.070699 23318 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38633
I20260924 03:16:08.070930 23318 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:08.071657 23318 heartbeater.cc:511] Master 127.14.227.190:38633 requested a full tablet report, sending...
I20260924 03:16:08.072924 22833 ts_manager.cc:194] Registered new tserver with Master: 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783)
I20260924 03:16:08.073693 22833 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:41053
I20260924 03:16:08.077488 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.061796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:43993 (local address 127.14.227.190:38633)
0924 03:16:08.062049 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.062057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.065074 (+ 3017us) server_negotiation.cc:407] Connection header received
0924 03:16:08.065307 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.065311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.065377 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.065474 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:08.066667 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.067919 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.069344 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.070019 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.070150 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.071985 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:08.071992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:08.071996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:08.072033 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:08.074979 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:08.075631 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:08.075638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:08.075644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:08.075774 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:08.076137 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:08.076142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:08.076145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:08.076396 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:08.076553 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.076926 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.077156 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:16:08.077549 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.061689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38633 (local address 127.14.227.131:43993)
0924 03:16:08.064969 (+ 3280us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.064992 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.065182 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.065497 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.065502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.066072 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:08.066494 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.066507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.068047 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.068053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.069181 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.069202 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.070195 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.070201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.070362 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.070915 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:08.070934 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:08.071820 (+ 886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:08.075106 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:08.075112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:08.075116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:08.075516 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:08.075881 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:08.075887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:08.075891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:08.076021 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:08.076511 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:08.076516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:08.076694 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.076833 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.077136 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3072,"thread_start_us":2909,"threads_started":1}
I20260924 03:16:08.078171 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 23183
I20260924 03:16:08.078344 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:16:08.078872 15246 external_mini_cluster.cc:1428] Setting key e572b4bd4d16d20027b6ea7be66e8d54
I20260924 03:16:08.081689 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 03:16:08 dist-test-slave-vd0p krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 03:16:08 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:08 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:16:08.126323 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:08.116521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.14.227.190:38633)
0924 03:16:08.116700 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.116705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.116956 (+ 251us) server_negotiation.cc:407] Connection header received
0924 03:16:08.117101 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.117105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.117169 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.117257 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:08.118555 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.119256 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.120398 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.120798 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.120885 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.126087 (+ 5202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52020: BlockingRecv error: recv got EOF from 127.0.0.1:52020 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 03:16:08 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:16:08.142071 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.129035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.14.227.190:38633)
0924 03:16:08.129242 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.129251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.129503 (+ 252us) server_negotiation.cc:407] Connection header received
0924 03:16:08.129691 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.129695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.129777 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.129878 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:08.130956 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.131622 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.132862 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.133293 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.133424 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.136973 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:08.136978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:08.136981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:08.137011 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:08.139545 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:08.140195 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:08.140200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:08.140204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:08.140322 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:08.140677 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:08.140683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:08.140687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:08.140953 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:08.141221 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.141500 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.141697 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:16:08.145697 22833 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:16:08.152576 22833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:16:08.191002 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.164367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.14.227.130:32981)
0924 03:16:08.182059 (+ 17692us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.182068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.183075 (+ 1007us) server_negotiation.cc:407] Connection header received
0924 03:16:08.183383 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.183390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.183597 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.183782 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:08.185109 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.186453 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.188678 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.189838 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.190031 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.190206 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.190500 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.190668 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17534,"thread_start_us":17406,"threads_started":1}
I20260924 03:16:08.192145 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.164071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.14.227.131:32783)
0924 03:16:08.183191 (+ 19120us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.183199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.183230 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:16:08.183332 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.183339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.183607 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.183795 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:08.184942 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.186645 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.189268 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.190639 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.190819 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.190970 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.191651 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.191822 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18941,"thread_start_us":19848,"threads_started":1}
I20260924 03:16:08.192237 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.164046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:32981 (local address 127.0.0.1:53000)
0924 03:16:08.182946 (+ 18900us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.182973 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.183181 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.184296 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.184301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.184341 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:08.184952 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.184965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.186607 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.186613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.188530 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.188541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.190042 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.190048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.190244 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.190299 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.190435 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.191703 (+ 1268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18753,"mutex_wait_us":1,"thread_start_us":17982,"threads_started":1}
I20260924 03:16:08.192405 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.166757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:40689 (local address 127.0.0.1:56266)
0924 03:16:08.184621 (+ 17864us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.184648 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.184784 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.185509 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.185514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.185538 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:08.185841 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.185853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.187561 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.187568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.189571 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.189581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.191591 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.191597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.191762 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.191801 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.191899 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.192013 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17720,"mutex_wait_us":2,"thread_start_us":15903,"threads_started":1}
I20260924 03:16:08.192464 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.163816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:32783 (local address 127.0.0.1:58062)
0924 03:16:08.181119 (+ 17303us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.181181 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.181422 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.184028 (+ 2606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.184035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.184066 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:08.184662 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.184679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.186797 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.186802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.189098 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.189112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.191307 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.191316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.191459 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.191481 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.191608 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.191735 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16959,"mutex_wait_us":32,"thread_start_us":20456,"threads_started":1}
I20260924 03:16:08.192991 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.167294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.14.227.129:40689)
0924 03:16:08.182502 (+ 15208us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.182511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.184756 (+ 2245us) server_negotiation.cc:407] Connection header received
0924 03:16:08.185021 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.185027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.185318 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.185496 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:08.186049 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.187443 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.189710 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.191112 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.191280 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.191444 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.192517 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.192710 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15064,"thread_start_us":14945,"threads_started":1}
I20260924 03:16:08.196015 23109 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee4ce284baf4b9f951b40376d07bb62 (DEFAULT_TABLE table=test-table [id=c9e8983b4eb642bcb5c1092cd2d1a8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:08.196010 23248 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee4ce284baf4b9f951b40376d07bb62 (DEFAULT_TABLE table=test-table [id=c9e8983b4eb642bcb5c1092cd2d1a8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:08.197286 23248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee4ce284baf4b9f951b40376d07bb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.197288 23109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee4ce284baf4b9f951b40376d07bb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.205180 22970 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee4ce284baf4b9f951b40376d07bb62 (DEFAULT_TABLE table=test-table [id=c9e8983b4eb642bcb5c1092cd2d1a8ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:08.206570 22970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee4ce284baf4b9f951b40376d07bb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.208871 23342 tablet_bootstrap.cc:492] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Bootstrap starting.
I20260924 03:16:08.211815 23342 tablet_bootstrap.cc:654] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.212785 23342 log.cc:824] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:08.215544 23342 tablet_bootstrap.cc:492] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: No bootstrap required, opened a new log
I20260924 03:16:08.215809 23342 ts_tablet_manager.cc:1397] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 03:16:08.219748 23345 tablet_bootstrap.cc:492] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Bootstrap starting.
I20260924 03:16:08.219921 23343 tablet_bootstrap.cc:492] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Bootstrap starting.
I20260924 03:16:08.222554 23343 tablet_bootstrap.cc:654] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.223018 23345 tablet_bootstrap.cc:654] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.223716 23343 log.cc:824] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:08.224184 23345 log.cc:824] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:08.224249 23342 raft_consensus.cc:348] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.224789 23342 raft_consensus.cc:374] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.224972 23342 raft_consensus.cc:729] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8066f0d4e34e36a2cd9d47cb3c73dd, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.225724 23342 consensus_queue.cc:260] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.227083 23179 heartbeater.cc:503] Master 127.14.227.190:38633 was elected leader, sending a full tablet report...
I20260924 03:16:08.227912 23342 ts_tablet_manager.cc:1428] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260924 03:16:08.228689 23345 tablet_bootstrap.cc:492] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: No bootstrap required, opened a new log
I20260924 03:16:08.228713 23343 tablet_bootstrap.cc:492] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: No bootstrap required, opened a new log
I20260924 03:16:08.228946 23345 ts_tablet_manager.cc:1397] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260924 03:16:08.228962 23343 ts_tablet_manager.cc:1397] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20260924 03:16:08.236519 23343 raft_consensus.cc:348] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.236855 23343 raft_consensus.cc:374] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.236963 23343 raft_consensus.cc:729] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943d37a1703465093f902bd7289e278, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.236784 23345 raft_consensus.cc:348] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.237236 23345 raft_consensus.cc:374] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.237341 23345 raft_consensus.cc:729] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f5bf792d454d54b85ec47d4715dc54, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.237628 23343 consensus_queue.cc:260] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.237867 23345 consensus_queue.cc:260] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.238756 23318 heartbeater.cc:503] Master 127.14.227.190:38633 was elected leader, sending a full tablet report...
I20260924 03:16:08.239307 23343 ts_tablet_manager.cc:1428] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 03:16:08.240566 23345 ts_tablet_manager.cc:1428] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260924 03:16:08.306581 23041 tablet.cc:2369] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:16:08.320683 23319 tablet.cc:2369] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:16:08.332387 23180 tablet.cc:2369] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:16:08.579033 23350 raft_consensus.cc:482] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:16:08.579346 23350 raft_consensus.cc:504] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.584678 23350 leader_election.cc:288] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783), cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:08.591292 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.581773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:34835 (local address 127.14.227.130:32981)
0924 03:16:08.582001 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.582011 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.583158 (+ 1147us) server_negotiation.cc:407] Connection header received
0924 03:16:08.583500 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.583509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.583604 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.583694 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:08.584609 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.586182 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.588922 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.590200 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.590353 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.590425 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.590903 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.591049 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:16:08.591327 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.581577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:32981 (local address 127.14.227.129:34835)
0924 03:16:08.583021 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.583054 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.583307 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.583946 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.583953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.583984 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:08.584420 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.584435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.586462 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.586470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.588772 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.588786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.590557 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.590563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.590715 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.590745 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.590857 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.590987 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"thread_start_us":2870,"threads_started":1}
I20260924 03:16:08.592356 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee4ce284baf4b9f951b40376d07bb62" candidate_uuid: "64f5bf792d454d54b85ec47d4715dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" is_pre_election: true
I20260924 03:16:08.592620 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.582254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:35381 (local address 127.14.227.131:32783)
0924 03:16:08.582471 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.582479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.586517 (+ 4038us) server_negotiation.cc:407] Connection header received
0924 03:16:08.586712 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.586717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.586790 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.586911 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:08.587766 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.588743 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.590338 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.591255 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.591403 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.591467 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.592255 (+ 788us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.592372 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:16:08.592813 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.581760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:32783 (local address 127.14.227.129:35381)
0924 03:16:08.586425 (+ 4665us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.586442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.586567 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.587076 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.587086 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.587109 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:08.587576 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.587594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.588917 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.588921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.590191 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.590203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.592007 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.592011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.592110 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.592130 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.592223 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.592326 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4539,"thread_start_us":4398,"threads_started":1}
I20260924 03:16:08.592921 23133 raft_consensus.cc:2454] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f5bf792d454d54b85ec47d4715dc54 in term 0.
I20260924 03:16:08.593537 22923 leader_election.cc:302] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f5bf792d454d54b85ec47d4715dc54, cd8066f0d4e34e36a2cd9d47cb3c73dd; no voters:
I20260924 03:16:08.593811 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee4ce284baf4b9f951b40376d07bb62" candidate_uuid: "64f5bf792d454d54b85ec47d4715dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943d37a1703465093f902bd7289e278" is_pre_election: true
I20260924 03:16:08.594000 23350 raft_consensus.cc:2781] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:16:08.594161 23350 raft_consensus.cc:482] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:16:08.594271 23350 raft_consensus.cc:3037] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:08.594312 23268 raft_consensus.cc:2454] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f5bf792d454d54b85ec47d4715dc54 in term 0.
I20260924 03:16:08.596948 23350 raft_consensus.cc:504] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.597479 23350 leader_election.cc:288] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 election: Requested vote from peers 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783), cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:08.597883 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee4ce284baf4b9f951b40376d07bb62" candidate_uuid: "64f5bf792d454d54b85ec47d4715dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943d37a1703465093f902bd7289e278"
I20260924 03:16:08.597997 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee4ce284baf4b9f951b40376d07bb62" candidate_uuid: "64f5bf792d454d54b85ec47d4715dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd"
I20260924 03:16:08.598116 23268 raft_consensus.cc:3037] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:08.598222 23133 raft_consensus.cc:3037] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:08.600009 23268 raft_consensus.cc:2454] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f5bf792d454d54b85ec47d4715dc54 in term 1.
I20260924 03:16:08.600211 23133 raft_consensus.cc:2454] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f5bf792d454d54b85ec47d4715dc54 in term 1.
I20260924 03:16:08.600383 22921 leader_election.cc:302] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3943d37a1703465093f902bd7289e278, 64f5bf792d454d54b85ec47d4715dc54; no voters:
I20260924 03:16:08.600674 23350 raft_consensus.cc:2781] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:08.601056 23350 raft_consensus.cc:686] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 LEADER]: Becoming Leader. State: Replica: 64f5bf792d454d54b85ec47d4715dc54, State: Running, Role: LEADER
I20260924 03:16:08.601483 23350 consensus_queue.cc:237] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } }
I20260924 03:16:08.606145 22833 catalog_manager.cc:5495] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "64f5bf792d454d54b85ec47d4715dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } health_report { overall_health: HEALTHY } } }
I20260924 03:16:08.647500 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.642431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.14.227.129:40689)
0924 03:16:08.642639 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.642644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.643002 (+ 358us) server_negotiation.cc:407] Connection header received
0924 03:16:08.643174 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.643179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.643252 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.643379 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:08.643754 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.644428 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.645652 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.646116 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.646217 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.646561 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:08.647062 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:08.647167 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.647248 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.647307 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:16:08.655267 23133 raft_consensus.cc:1261] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Refusing update from remote peer 64f5bf792d454d54b85ec47d4715dc54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:08.655326 23268 raft_consensus.cc:1261] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Refusing update from remote peer 64f5bf792d454d54b85ec47d4715dc54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:08.656327 23353 consensus_queue.cc:1035] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:08.656759 23350 consensus_queue.cc:1035] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:08.670552 23358 mvcc.cc:205] Tried to move back new op lower bound from 7332740172402970624 to 7332740172195684352. Current Snapshot: MvccSnapshot[applied={T|T < 7332740172402970624}]
I20260924 03:16:08.680410 22832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:08.680658 22832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:08.684068 22832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: Sending DeleteTablet for 3 replicas of tablet 7ee4ce284baf4b9f951b40376d07bb62
I20260924 03:16:08.685127 23248 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee4ce284baf4b9f951b40376d07bb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:08 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260924 03:16:08.685560 22970 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee4ce284baf4b9f951b40376d07bb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:08 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56266
I20260924 03:16:08.685582 23109 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee4ce284baf4b9f951b40376d07bb62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:08 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:53000
I20260924 03:16:08.686167 23367 tablet_replica.cc:330] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: stopping tablet replica
I20260924 03:16:08.686702 23367 raft_consensus.cc:2229] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:08.687337 23367 raft_consensus.cc:2258] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:08.687426 22832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:16:08.688302 22832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:16:08.690078 23367 ts_tablet_manager.cc:1905] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:08.697700 23108 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1d4f32e84478fbf1449509bdeb789 (DEFAULT_TABLE table=test-table [id=f846d550f6fc4bbab2bc56929051815a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:08.698328 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1d4f32e84478fbf1449509bdeb789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.698974 22969 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1d4f32e84478fbf1449509bdeb789 (DEFAULT_TABLE table=test-table [id=f846d550f6fc4bbab2bc56929051815a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:08.699683 22969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1d4f32e84478fbf1449509bdeb789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.700841 23248 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1d4f32e84478fbf1449509bdeb789 (DEFAULT_TABLE table=test-table [id=f846d550f6fc4bbab2bc56929051815a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:08.701356 23248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1d4f32e84478fbf1449509bdeb789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.704519 23369 tablet_replica.cc:330] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: stopping tablet replica
I20260924 03:16:08.704998 23369 raft_consensus.cc:2229] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:08.705467 23369 raft_consensus.cc:2258] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:08.706291 23342 tablet_bootstrap.cc:492] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Bootstrap starting.
I20260924 03:16:08.706277 23368 tablet_replica.cc:330] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: stopping tablet replica
I20260924 03:16:08.706655 23368 raft_consensus.cc:2229] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:16:08.707415 23368 raft_consensus.cc:2258] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:08.707832 23342 tablet_bootstrap.cc:654] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.709779 23368 ts_tablet_manager.cc:1905] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:08.714040 23367 ts_tablet_manager.cc:1918] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:08.715739 23367 log.cc:1196] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/7ee4ce284baf4b9f951b40376d07bb62
I20260924 03:16:08.716499 23367 ts_tablet_manager.cc:1939] T 7ee4ce284baf4b9f951b40376d07bb62 P 3943d37a1703465093f902bd7289e278: Deleting consensus metadata
I20260924 03:16:08.717907 23368 ts_tablet_manager.cc:1918] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:08.718358 23368 log.cc:1196] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/7ee4ce284baf4b9f951b40376d07bb62
I20260924 03:16:08.718897 23368 ts_tablet_manager.cc:1939] T 7ee4ce284baf4b9f951b40376d07bb62 P 64f5bf792d454d54b85ec47d4715dc54: Deleting consensus metadata
I20260924 03:16:08.719451 23342 tablet_bootstrap.cc:492] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: No bootstrap required, opened a new log
I20260924 03:16:08.719627 23342 ts_tablet_manager.cc:1397] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20260924 03:16:08.720432 23342 raft_consensus.cc:348] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.720585 22817 catalog_manager.cc:4841] TS 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689): tablet 7ee4ce284baf4b9f951b40376d07bb62 (table test-table [id=c9e8983b4eb642bcb5c1092cd2d1a8ca]) successfully deleted
I20260924 03:16:08.720793 23342 raft_consensus.cc:374] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.720952 23342 raft_consensus.cc:729] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8066f0d4e34e36a2cd9d47cb3c73dd, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.721140 22818 catalog_manager.cc:4841] TS 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783): tablet 7ee4ce284baf4b9f951b40376d07bb62 (table test-table [id=c9e8983b4eb642bcb5c1092cd2d1a8ca]) successfully deleted
I20260924 03:16:08.721379 23342 consensus_queue.cc:260] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.721557 23345 tablet_bootstrap.cc:492] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: Bootstrap starting.
I20260924 03:16:08.722844 23342 ts_tablet_manager.cc:1428] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 03:16:08.723321 23369 ts_tablet_manager.cc:1905] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:08.723412 23345 tablet_bootstrap.cc:654] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.728498 23369 ts_tablet_manager.cc:1918] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:08.728660 23369 log.cc:1196] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/7ee4ce284baf4b9f951b40376d07bb62
I20260924 03:16:08.729046 23343 tablet_bootstrap.cc:492] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: Bootstrap starting.
I20260924 03:16:08.729071 23369 ts_tablet_manager.cc:1939] T 7ee4ce284baf4b9f951b40376d07bb62 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting consensus metadata
I20260924 03:16:08.730360 22820 catalog_manager.cc:4841] TS cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981): tablet 7ee4ce284baf4b9f951b40376d07bb62 (table test-table [id=c9e8983b4eb642bcb5c1092cd2d1a8ca]) successfully deleted
I20260924 03:16:08.730746 23343 tablet_bootstrap.cc:654] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.731642 23345 tablet_bootstrap.cc:492] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: No bootstrap required, opened a new log
I20260924 03:16:08.731842 23345 ts_tablet_manager.cc:1397] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260924 03:16:08.732698 23345 raft_consensus.cc:348] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.732928 23345 raft_consensus.cc:374] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.733033 23345 raft_consensus.cc:729] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f5bf792d454d54b85ec47d4715dc54, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.733043 23343 tablet_bootstrap.cc:492] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: No bootstrap required, opened a new log
I20260924 03:16:08.733345 23343 ts_tablet_manager.cc:1397] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 03:16:08.733289 23345 consensus_queue.cc:260] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.733523 23362 raft_consensus.cc:482] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:16:08.733928 23345 ts_tablet_manager.cc:1428] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:16:08.733772 23362 raft_consensus.cc:504] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.734710 23362 leader_election.cc:288] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981), 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783)
I20260924 03:16:08.735112 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac1d4f32e84478fbf1449509bdeb789" candidate_uuid: "64f5bf792d454d54b85ec47d4715dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" is_pre_election: true
I20260924 03:16:08.735357 23133 raft_consensus.cc:2454] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f5bf792d454d54b85ec47d4715dc54 in term 0.
I20260924 03:16:08.735383 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac1d4f32e84478fbf1449509bdeb789" candidate_uuid: "64f5bf792d454d54b85ec47d4715dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943d37a1703465093f902bd7289e278" is_pre_election: true
I20260924 03:16:08.735841 22923 leader_election.cc:302] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f5bf792d454d54b85ec47d4715dc54, cd8066f0d4e34e36a2cd9d47cb3c73dd; no voters:
I20260924 03:16:08.736112 23362 raft_consensus.cc:2781] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:16:08.736270 23362 raft_consensus.cc:482] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:16:08.736375 23362 raft_consensus.cc:3037] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:08.739705 23362 raft_consensus.cc:504] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.740406 23362 leader_election.cc:288] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 election: Requested vote from peers cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981), 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783)
W20260924 03:16:08.736120 22921 leader_election.cc:341] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783): Illegal state: must be running to vote when last-logged opid is not known
I20260924 03:16:08.740900 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac1d4f32e84478fbf1449509bdeb789" candidate_uuid: "64f5bf792d454d54b85ec47d4715dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd"
I20260924 03:16:08.741097 23133 raft_consensus.cc:3037] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:08.741410 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac1d4f32e84478fbf1449509bdeb789" candidate_uuid: "64f5bf792d454d54b85ec47d4715dc54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943d37a1703465093f902bd7289e278"
W20260924 03:16:08.742413 22921 leader_election.cc:341] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783): Illegal state: must be running to vote when last-logged opid is not known
I20260924 03:16:08.742352 23343 raft_consensus.cc:348] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.742578 23343 raft_consensus.cc:374] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.742677 23343 raft_consensus.cc:729] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943d37a1703465093f902bd7289e278, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.742964 23343 consensus_queue.cc:260] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.743496 23133 raft_consensus.cc:2454] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f5bf792d454d54b85ec47d4715dc54 in term 1.
I20260924 03:16:08.743579 23343 ts_tablet_manager.cc:1428] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20260924 03:16:08.744052 22923 leader_election.cc:302] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f5bf792d454d54b85ec47d4715dc54, cd8066f0d4e34e36a2cd9d47cb3c73dd; no voters: 3943d37a1703465093f902bd7289e278
I20260924 03:16:08.744366 23362 raft_consensus.cc:2781] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:08.744606 23362 raft_consensus.cc:686] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 LEADER]: Becoming Leader. State: Replica: 64f5bf792d454d54b85ec47d4715dc54, State: Running, Role: LEADER
I20260924 03:16:08.744957 23362 consensus_queue.cc:237] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.748134 22833 catalog_manager.cc:5495] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "64f5bf792d454d54b85ec47d4715dc54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } health_report { overall_health: UNKNOWN } } }
Sep 24 03:16:08 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219766, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.190@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:16:08.790365 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.776268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.14.227.190:38633)
0924 03:16:08.777222 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.777227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.777251 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:16:08.777289 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.777292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.777339 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.777447 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:08.778577 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.779302 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.780559 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.781158 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.781308 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.784951 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:08.784964 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:08.784968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:08.784993 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:08.787563 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:08.788462 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:08.788465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:08.788467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:08.788573 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:08.789024 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:08.789027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:08.789029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:08.789278 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:08.789378 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.789873 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.790083 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":825,"thread_start_us":642,"threads_started":1}
I20260924 03:16:08.790786 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.776219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38633 (local address 127.0.0.1:52032)
0924 03:16:08.776892 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.776922 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.777104 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.777473 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.777479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.778217 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:08.778413 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.778422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.779427 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.779433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.780424 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.780435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.781323 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.781328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.781538 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.782310 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:08.782327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:08.784757 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:08.787698 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:08.787711 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:08.787720 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:08.788342 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:08.788682 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:08.788688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:08.788692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:08.788908 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:08.789400 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:08.789409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:08.789615 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.789810 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.790253 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":114,"thread_start_us":395,"threads_started":1}
I20260924 03:16:08.796195 22833 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:52032:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 03:16:08.797341 22833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:16:08.804427 23108 tablet_service.cc:1467] Processing CreateTablet for tablet 16817bf8b38e4c94bd45f69bd8168f6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15ba255cf74f408f9f2b580c49455232]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:16:08.804442 22969 tablet_service.cc:1467] Processing CreateTablet for tablet 16817bf8b38e4c94bd45f69bd8168f6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15ba255cf74f408f9f2b580c49455232]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:16:08.804937 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16817bf8b38e4c94bd45f69bd8168f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.805014 22969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16817bf8b38e4c94bd45f69bd8168f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.810303 23342 tablet_bootstrap.cc:492] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: Bootstrap starting.
I20260924 03:16:08.811594 23248 tablet_service.cc:1467] Processing CreateTablet for tablet 16817bf8b38e4c94bd45f69bd8168f6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15ba255cf74f408f9f2b580c49455232]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 03:16:08.811969 23342 tablet_bootstrap.cc:654] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.812153 23248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16817bf8b38e4c94bd45f69bd8168f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:08.813033 23345 tablet_bootstrap.cc:492] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: Bootstrap starting.
I20260924 03:16:08.814888 23345 tablet_bootstrap.cc:654] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.821673 23343 tablet_bootstrap.cc:492] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: Bootstrap starting.
I20260924 03:16:08.823263 23342 tablet_bootstrap.cc:492] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: No bootstrap required, opened a new log
I20260924 03:16:08.823553 23343 tablet_bootstrap.cc:654] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:08.823807 23342 ts_tablet_manager.cc:1397] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260924 03:16:08.824961 23342 raft_consensus.cc:348] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.825260 23342 raft_consensus.cc:374] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.825372 23342 raft_consensus.cc:729] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8066f0d4e34e36a2cd9d47cb3c73dd, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.825798 23343 tablet_bootstrap.cc:492] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: No bootstrap required, opened a new log
I20260924 03:16:08.825711 23342 consensus_queue.cc:260] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.826020 23343 ts_tablet_manager.cc:1397] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260924 03:16:08.826263 23348 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.826642 23348 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:16:08.827294 23343 raft_consensus.cc:348] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.827598 23343 raft_consensus.cc:374] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.827715 23343 raft_consensus.cc:729] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943d37a1703465093f902bd7289e278, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.827991 23343 consensus_queue.cc:260] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.828423 23349 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.828771 23349 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:16:08.829622 23343 ts_tablet_manager.cc:1428] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:16:08.830309 23342 ts_tablet_manager.cc:1428] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260924 03:16:08.830397 23345 tablet_bootstrap.cc:492] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: No bootstrap required, opened a new log
I20260924 03:16:08.830576 23345 ts_tablet_manager.cc:1397] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20260924 03:16:08.831363 23345 raft_consensus.cc:348] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.831624 23345 raft_consensus.cc:374] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:08.831708 23345 raft_consensus.cc:729] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f5bf792d454d54b85ec47d4715dc54, State: Initialized, Role: FOLLOWER
I20260924 03:16:08.831979 23345 consensus_queue.cc:260] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.832401 23362 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.832664 23362 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:16:08.833637 23345 ts_tablet_manager.cc:1428] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:16:08.904539 23349 raft_consensus.cc:482] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:16:08.904781 23349 raft_consensus.cc:504] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.906301 23349 leader_election.cc:288] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689), cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:08.914307 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.906754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:39943 (local address 127.14.227.129:40689)
0924 03:16:08.907068 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.907075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.907727 (+ 652us) server_negotiation.cc:407] Connection header received
0924 03:16:08.907957 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.907962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.908069 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.908162 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:08.908933 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.910194 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.912409 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.913222 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.913406 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.913480 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.913811 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.913950 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260924 03:16:08.914589 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.906607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:32981 (local address 127.14.227.131:34583)
0924 03:16:08.907338 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.907361 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.907499 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.907972 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.907976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.907995 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:08.908380 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.908391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.909991 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.909996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.912068 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.912082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.913800 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.913805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.914011 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.914040 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.914142 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.914263 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":840,"threads_started":1}
I20260924 03:16:08.914636 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.907222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:34583 (local address 127.14.227.130:32981)
0924 03:16:08.907393 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.907400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.907445 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:16:08.907726 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.907733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.907813 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.907914 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:08.908540 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.909868 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.912239 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.913575 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.913746 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.913826 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.914190 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.914330 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:16:08.915428 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.906611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:40689 (local address 127.14.227.131:39943)
0924 03:16:08.907634 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.907650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.907831 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.908214 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.908220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.908252 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:08.908725 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.908736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.910331 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.910335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.912277 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.912286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.913406 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.913413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.913598 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.913622 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.914952 (+ 1330us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.915081 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":25,"thread_start_us":372,"threads_started":1}
I20260924 03:16:08.915336 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16817bf8b38e4c94bd45f69bd8168f6b" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" is_pre_election: true
I20260924 03:16:08.915673 23133 raft_consensus.cc:2454] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 0.
I20260924 03:16:08.916157 23202 leader_election.cc:302] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3943d37a1703465093f902bd7289e278, cd8066f0d4e34e36a2cd9d47cb3c73dd; no voters:
I20260924 03:16:08.916558 23349 raft_consensus.cc:2781] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:16:08.916360 22990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16817bf8b38e4c94bd45f69bd8168f6b" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f5bf792d454d54b85ec47d4715dc54" is_pre_election: true
I20260924 03:16:08.916697 23349 raft_consensus.cc:482] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:16:08.916800 23349 raft_consensus.cc:3037] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:08.916851 22990 raft_consensus.cc:2454] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 0.
I20260924 03:16:08.919384 23349 raft_consensus.cc:504] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.919960 23349 leader_election.cc:288] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 election: Requested vote from peers 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689), cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:08.920413 22990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16817bf8b38e4c94bd45f69bd8168f6b" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f5bf792d454d54b85ec47d4715dc54"
I20260924 03:16:08.920416 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16817bf8b38e4c94bd45f69bd8168f6b" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd"
I20260924 03:16:08.920606 22990 raft_consensus.cc:3037] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:08.920631 23133 raft_consensus.cc:3037] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:08.923123 22990 raft_consensus.cc:2454] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 1.
I20260924 03:16:08.923508 23199 leader_election.cc:302] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3943d37a1703465093f902bd7289e278, 64f5bf792d454d54b85ec47d4715dc54; no voters:
I20260924 03:16:08.923748 23133 raft_consensus.cc:2454] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 1.
I20260924 03:16:08.923830 23349 raft_consensus.cc:2781] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:08.924366 23349 raft_consensus.cc:686] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [term 1 LEADER]: Becoming Leader. State: Replica: 3943d37a1703465093f902bd7289e278, State: Running, Role: LEADER
I20260924 03:16:08.924947 23349 consensus_queue.cc:237] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:08.926914 23386 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: TxnStatusTablet state changed. Reason: New leader 3943d37a1703465093f902bd7289e278. Latest consensus state: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.927327 23386 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:16:08.928179 23388 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 03:16:08.928385 23388 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 03:16:08.928786 22832 catalog_manager.cc:5495] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3943d37a1703465093f902bd7289e278 (127.14.227.131). New cstate: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } health_report { overall_health: HEALTHY } } }
I20260924 03:16:08.958869 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.951292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.14.227.131:32783)
0924 03:16:08.951468 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:08.951476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:08.951794 (+ 318us) server_negotiation.cc:407] Connection header received
0924 03:16:08.952122 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:08.952128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:08.952235 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:08.952382 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:08.952775 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.954098 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.956188 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.956847 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.956990 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.957545 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:08.958204 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:08.958344 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:08.958479 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 03:16:08.958563 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:16:08.959223 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:08.951313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:32783 (local address 127.0.0.1:58066)
0924 03:16:08.951688 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:08.951709 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:08.951863 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:08.952436 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:08.952441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:08.952454 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:16:08.952636 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.952647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.954251 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.954257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.956053 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:08.956069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:08.957029 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:08.957036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:08.957284 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:08.957352 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:16:08.958325 (+ 973us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:16:08.958347 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:16:08.958447 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:16:08.958648 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260924 03:16:08.963620 23133 raft_consensus.cc:1261] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Refusing update from remote peer 3943d37a1703465093f902bd7289e278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:08.964105 22990 raft_consensus.cc:1261] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Refusing update from remote peer 3943d37a1703465093f902bd7289e278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:08.964408 23349 consensus_queue.cc:1035] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:08.965011 23349 consensus_queue.cc:1035] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:08.978087 23362 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: TxnStatusTablet state changed. Reason: New leader 3943d37a1703465093f902bd7289e278. Latest consensus state: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.978449 23362 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:16:08.978790 23348 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: TxnStatusTablet state changed. Reason: New leader 3943d37a1703465093f902bd7289e278. Latest consensus state: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.979070 23348 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:16:08.981043 23386 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.981269 23386 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:16:08.981770 23348 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.982008 23348 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P cd8066f0d4e34e36a2cd9d47cb3c73dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:16:08.983136 23359 mvcc.cc:205] Tried to move back new op lower bound from 7332740173667397632 to 7332740173520031744. Current Snapshot: MvccSnapshot[applied={T|T < 7332740173667397632 or (T in {7332740173667397632})}]
I20260924 03:16:08.985204 23386 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.985458 23386 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P 3943d37a1703465093f902bd7289e278: This TxnStatusTablet replica's current role is: LEADER
I20260924 03:16:08.986901 23362 tablet_replica.cc:439] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } } }
I20260924 03:16:08.987226 23362 tablet_replica.cc:442] T 16817bf8b38e4c94bd45f69bd8168f6b P 64f5bf792d454d54b85ec47d4715dc54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 03:16:09.009864 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.002023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:38351 (local address 127.14.227.131:32783)
0924 03:16:09.002240 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:09.002247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:09.003120 (+ 873us) server_negotiation.cc:407] Connection header received
0924 03:16:09.003316 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:09.003321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:09.003410 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:09.003516 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:09.004218 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.005353 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.007667 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.008404 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.008525 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.009105 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:09.009233 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:09.009321 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:09.009489 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 03:16:09.009569 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:16:09.010113 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.002165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:32783 (local address 127.14.227.129:38351)
0924 03:16:09.002984 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:09.003013 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:09.003181 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:09.003545 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:09.003549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:09.003562 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:16:09.004037 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:09.004054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.005500 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.005506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:09.007524 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:09.007541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.008574 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.008580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:09.008810 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.008882 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:16:09.009345 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:16:09.009358 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 03:16:09.009460 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:16:09.009606 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":380,"threads_started":1}
I20260924 03:16:09.024694 23268 raft_consensus.cc:3037] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:09.025125 23130 raft_consensus.cc:1261] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Refusing update from remote peer 64f5bf792d454d54b85ec47d4715dc54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:09.025763 23350 consensus_queue.cc:1035] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:09.027595 23268 raft_consensus.cc:1261] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Refusing update from remote peer 64f5bf792d454d54b85ec47d4715dc54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:09.028221 23350 consensus_queue.cc:1035] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:09.032934 23357 mvcc.cc:205] Tried to move back new op lower bound from 7332740173918662656 to 7332740172782432256. Current Snapshot: MvccSnapshot[applied={T|T < 7332740173918662656}]
I20260924 03:16:09.103263 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.078266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:48965 (local address 127.14.227.129:40689)
0924 03:16:09.078468 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:09.078475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:09.093884 (+ 15409us) server_negotiation.cc:407] Connection header received
0924 03:16:09.094201 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:09.094207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:09.094297 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:09.094359 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:09.095219 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.096388 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.100828 (+ 4440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.101517 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.101634 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.102400 (+ 766us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:09.102531 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:09.102592 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:09.102857 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 03:16:09.102942 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:16:09.104417 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.068930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:40689 (local address 127.14.227.131:48965)
0924 03:16:09.093781 (+ 24851us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:09.093804 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:09.094030 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:09.094557 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:09.094562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:09.094574 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:16:09.095005 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:09.095017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.098666 (+ 3649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.098673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:09.100668 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:09.100683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.101861 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.101867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:09.102101 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.102202 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:16:09.102672 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:16:09.102681 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:16:09.103722 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0924 03:16:09.103876 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24606,"thread_start_us":24429,"threads_started":1}
I20260924 03:16:09.176138 22832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:09.176379 22832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:09.179384 22832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: Sending DeleteTablet for 3 replicas of tablet 1ac1d4f32e84478fbf1449509bdeb789
I20260924 03:16:09.180366 22969 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1d4f32e84478fbf1449509bdeb789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56266
I20260924 03:16:09.180526 23108 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1d4f32e84478fbf1449509bdeb789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:53000
I20260924 03:16:09.180876 23368 tablet_replica.cc:330] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: stopping tablet replica
I20260924 03:16:09.180989 23248 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1d4f32e84478fbf1449509bdeb789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260924 03:16:09.181114 23368 raft_consensus.cc:2229] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:16:09.181458 23368 raft_consensus.cc:2258] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.181766 23369 tablet_replica.cc:330] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: stopping tablet replica
I20260924 03:16:09.181991 23369 raft_consensus.cc:2229] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:09.182140 23367 tablet_replica.cc:330] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: stopping tablet replica
I20260924 03:16:09.182193 23369 raft_consensus.cc:2258] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.182523 23367 raft_consensus.cc:2229] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:09.182735 23367 raft_consensus.cc:2258] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.183460 23369 ts_tablet_manager.cc:1905] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:09.185989 23368 ts_tablet_manager.cc:1905] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:09.191395 23367 ts_tablet_manager.cc:1905] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:09.193499 23368 ts_tablet_manager.cc:1918] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:16:09.193684 23368 log.cc:1196] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/1ac1d4f32e84478fbf1449509bdeb789
I20260924 03:16:09.198982 23367 ts_tablet_manager.cc:1918] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:16:09.199928 23369 ts_tablet_manager.cc:1918] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:16:09.200174 23367 log.cc:1196] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/1ac1d4f32e84478fbf1449509bdeb789
I20260924 03:16:09.200654 23367 ts_tablet_manager.cc:1939] T 1ac1d4f32e84478fbf1449509bdeb789 P 3943d37a1703465093f902bd7289e278: Deleting consensus metadata
I20260924 03:16:09.200840 23369 log.cc:1196] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/1ac1d4f32e84478fbf1449509bdeb789
I20260924 03:16:09.201722 23369 ts_tablet_manager.cc:1939] T 1ac1d4f32e84478fbf1449509bdeb789 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting consensus metadata
I20260924 03:16:09.202402 23368 ts_tablet_manager.cc:1939] T 1ac1d4f32e84478fbf1449509bdeb789 P 64f5bf792d454d54b85ec47d4715dc54: Deleting consensus metadata
I20260924 03:16:09.203595 22820 catalog_manager.cc:4841] TS cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981): tablet 1ac1d4f32e84478fbf1449509bdeb789 (table test-table [id=f846d550f6fc4bbab2bc56929051815a]) successfully deleted
I20260924 03:16:09.203591 22817 catalog_manager.cc:4841] TS 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689): tablet 1ac1d4f32e84478fbf1449509bdeb789 (table test-table [id=f846d550f6fc4bbab2bc56929051815a]) successfully deleted
I20260924 03:16:09.204211 22818 catalog_manager.cc:4841] TS 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783): tablet 1ac1d4f32e84478fbf1449509bdeb789 (table test-table [id=f846d550f6fc4bbab2bc56929051815a]) successfully deleted
Sep 24 03:16:09 dist-test-slave-vd0p krb5kdc[22788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 03:16:09 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:09 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:16:09.263779 23380 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:09.250731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52034 (local address 127.14.227.190:38633)
0924 03:16:09.250936 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:09.250942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:09.251400 (+ 458us) server_negotiation.cc:407] Connection header received
0924 03:16:09.251712 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:09.251719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:09.251808 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:09.251926 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:09.253583 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.254795 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.256682 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.257385 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.257525 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.263532 (+ 6007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52034: BlockingRecv error: recv got EOF from 127.0.0.1:52034 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 03:16:09 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM
I20260924 03:16:09.286427 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.269313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.14.227.190:38633)
0924 03:16:09.269465 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:09.269471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:09.269892 (+ 421us) server_negotiation.cc:407] Connection header received
0924 03:16:09.270093 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:09.270099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:09.270195 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:09.270300 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:09.271692 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.272832 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.274606 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.275258 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.275384 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.279966 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:09.279972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:09.279978 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:09.280015 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:09.283472 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:09.284287 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:09.284295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:09.284300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:09.284460 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:09.284946 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:09.284950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:09.284953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:09.285311 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:09.285577 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:09.285854 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 03:16:09.286119 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:16:09.290393 22832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:16:09.291174 22832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:16:09.299358 23108 tablet_service.cc:1467] Processing CreateTablet for tablet c7cfd0f3042042aea47aca65f5cc0c11 (DEFAULT_TABLE table=test-table [id=a04b55abb13342ad83ffb8be4123596e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:09.299486 22969 tablet_service.cc:1467] Processing CreateTablet for tablet c7cfd0f3042042aea47aca65f5cc0c11 (DEFAULT_TABLE table=test-table [id=a04b55abb13342ad83ffb8be4123596e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:09.299870 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cfd0f3042042aea47aca65f5cc0c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:09.299988 22969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cfd0f3042042aea47aca65f5cc0c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:09.306752 23342 tablet_bootstrap.cc:492] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Bootstrap starting.
I20260924 03:16:09.308293 23342 tablet_bootstrap.cc:654] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:09.309764 23248 tablet_service.cc:1467] Processing CreateTablet for tablet c7cfd0f3042042aea47aca65f5cc0c11 (DEFAULT_TABLE table=test-table [id=a04b55abb13342ad83ffb8be4123596e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:09.310195 23248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cfd0f3042042aea47aca65f5cc0c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:09.316938 23345 tablet_bootstrap.cc:492] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: Bootstrap starting.
I20260924 03:16:09.317044 23343 tablet_bootstrap.cc:492] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: Bootstrap starting.
I20260924 03:16:09.318715 23343 tablet_bootstrap.cc:654] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:09.318723 23345 tablet_bootstrap.cc:654] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:09.331324 23345 tablet_bootstrap.cc:492] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: No bootstrap required, opened a new log
I20260924 03:16:09.331358 23343 tablet_bootstrap.cc:492] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: No bootstrap required, opened a new log
I20260924 03:16:09.331489 23345 ts_tablet_manager.cc:1397] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260924 03:16:09.331523 23343 ts_tablet_manager.cc:1397] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.008s
I20260924 03:16:09.332386 23342 tablet_bootstrap.cc:492] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: No bootstrap required, opened a new log
I20260924 03:16:09.332304 23345 raft_consensus.cc:348] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.332566 23345 raft_consensus.cc:374] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:09.332669 23345 raft_consensus.cc:729] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f5bf792d454d54b85ec47d4715dc54, State: Initialized, Role: FOLLOWER
I20260924 03:16:09.332801 23342 ts_tablet_manager.cc:1397] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20260924 03:16:09.332929 23345 consensus_queue.cc:260] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.333318 23343 raft_consensus.cc:348] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.333578 23343 raft_consensus.cc:374] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:09.333706 23343 raft_consensus.cc:729] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943d37a1703465093f902bd7289e278, State: Initialized, Role: FOLLOWER
I20260924 03:16:09.333922 23342 raft_consensus.cc:348] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.334007 23343 consensus_queue.cc:260] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.334300 23342 raft_consensus.cc:374] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:09.334441 23342 raft_consensus.cc:729] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8066f0d4e34e36a2cd9d47cb3c73dd, State: Initialized, Role: FOLLOWER
I20260924 03:16:09.334688 23342 consensus_queue.cc:260] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.335881 23342 ts_tablet_manager.cc:1428] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 03:16:09.336450 23343 ts_tablet_manager.cc:1428] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 03:16:09.336954 23345 ts_tablet_manager.cc:1428] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 03:16:09.531414 23396 raft_consensus.cc:482] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:16:09.531616 23396 raft_consensus.cc:504] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.532191 23396 leader_election.cc:288] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689), cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:09.532689 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cfd0f3042042aea47aca65f5cc0c11" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" is_pre_election: true
I20260924 03:16:09.532691 22989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cfd0f3042042aea47aca65f5cc0c11" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f5bf792d454d54b85ec47d4715dc54" is_pre_election: true
I20260924 03:16:09.532948 22989 raft_consensus.cc:2454] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 0.
I20260924 03:16:09.532940 23130 raft_consensus.cc:2454] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 0.
I20260924 03:16:09.533382 23199 leader_election.cc:302] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3943d37a1703465093f902bd7289e278, 64f5bf792d454d54b85ec47d4715dc54; no voters:
I20260924 03:16:09.533666 23396 raft_consensus.cc:2781] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:16:09.533803 23396 raft_consensus.cc:482] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:16:09.533897 23396 raft_consensus.cc:3037] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:09.535854 23396 raft_consensus.cc:504] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.536417 23396 leader_election.cc:288] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 election: Requested vote from peers 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689), cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:09.536845 22989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cfd0f3042042aea47aca65f5cc0c11" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f5bf792d454d54b85ec47d4715dc54"
I20260924 03:16:09.536867 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cfd0f3042042aea47aca65f5cc0c11" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd"
I20260924 03:16:09.537011 22989 raft_consensus.cc:3037] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:09.537039 23130 raft_consensus.cc:3037] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:09.538980 23130 raft_consensus.cc:2454] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 1.
I20260924 03:16:09.538983 22989 raft_consensus.cc:2454] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 1.
I20260924 03:16:09.539368 23202 leader_election.cc:302] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3943d37a1703465093f902bd7289e278, 64f5bf792d454d54b85ec47d4715dc54, cd8066f0d4e34e36a2cd9d47cb3c73dd; no voters:
I20260924 03:16:09.539683 23396 raft_consensus.cc:2781] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:09.539849 23396 raft_consensus.cc:686] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 1 LEADER]: Becoming Leader. State: Replica: 3943d37a1703465093f902bd7289e278, State: Running, Role: LEADER
I20260924 03:16:09.540096 23396 consensus_queue.cc:237] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.542443 22833 catalog_manager.cc:5495] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3943d37a1703465093f902bd7289e278 (127.14.227.131). New cstate: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } health_report { overall_health: HEALTHY } } }
I20260924 03:16:09.568080 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.563278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.14.227.131:32783)
0924 03:16:09.563776 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:09.563784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:09.563801 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:16:09.563869 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:09.563872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:09.563930 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:09.564040 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:09.564411 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.565105 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.566486 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.566893 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.567031 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.567401 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:09.567498 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:09.567593 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:09.567739 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:16:09.567807 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":212,"threads_started":1}
I20260924 03:16:09.570709 23130 raft_consensus.cc:1261] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Refusing update from remote peer 3943d37a1703465093f902bd7289e278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:09.570739 22989 raft_consensus.cc:1261] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Refusing update from remote peer 3943d37a1703465093f902bd7289e278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:09.571269 23349 consensus_queue.cc:1035] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:09.571513 23396 consensus_queue.cc:1035] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:09.586175 22833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:09.586385 22833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:09.589370 22833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: Sending DeleteTablet for 3 replicas of tablet c7cfd0f3042042aea47aca65f5cc0c11
I20260924 03:16:09.590090 22969 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cfd0f3042042aea47aca65f5cc0c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56266
I20260924 03:16:09.590349 23368 tablet_replica.cc:330] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: stopping tablet replica
I20260924 03:16:09.590512 23368 raft_consensus.cc:2229] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:09.590723 23368 raft_consensus.cc:2258] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.591148 23108 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cfd0f3042042aea47aca65f5cc0c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:53000
I20260924 03:16:09.591730 23368 ts_tablet_manager.cc:1905] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:09.594158 23369 tablet_replica.cc:330] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: stopping tablet replica
I20260924 03:16:09.594337 23369 raft_consensus.cc:2229] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:09.594570 23369 raft_consensus.cc:2258] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.595551 23248 tablet_service.cc:1514] Processing DeleteTablet for tablet c7cfd0f3042042aea47aca65f5cc0c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260924 03:16:09.595801 23367 tablet_replica.cc:330] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: stopping tablet replica
I20260924 03:16:09.595973 23367 raft_consensus.cc:2229] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:16:09.596479 23367 raft_consensus.cc:2258] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.597318 22833 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 03:16:09.597587 23367 ts_tablet_manager.cc:1905] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 03:16:09.598193 22833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:16:09.599974 23368 ts_tablet_manager.cc:1918] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:09.600090 23368 log.cc:1196] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/c7cfd0f3042042aea47aca65f5cc0c11
I20260924 03:16:09.601346 23368 ts_tablet_manager.cc:1939] T c7cfd0f3042042aea47aca65f5cc0c11 P 64f5bf792d454d54b85ec47d4715dc54: Deleting consensus metadata
I20260924 03:16:09.603071 23369 ts_tablet_manager.cc:1905] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:09.604269 22817 catalog_manager.cc:4841] TS 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689): tablet c7cfd0f3042042aea47aca65f5cc0c11 (table test-table [id=a04b55abb13342ad83ffb8be4123596e]) successfully deleted
I20260924 03:16:09.605937 23108 tablet_service.cc:1467] Processing CreateTablet for tablet 6e46e30a4a424e80b9cfdec12b0875fc (DEFAULT_TABLE table=test-table [id=2ed61e6bb8af4cd88cb79a01e49bdabf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:09.606091 23248 tablet_service.cc:1467] Processing CreateTablet for tablet 6e46e30a4a424e80b9cfdec12b0875fc (DEFAULT_TABLE table=test-table [id=2ed61e6bb8af4cd88cb79a01e49bdabf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:09.606362 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e46e30a4a424e80b9cfdec12b0875fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:09.606473 23248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e46e30a4a424e80b9cfdec12b0875fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:09.606591 22969 tablet_service.cc:1467] Processing CreateTablet for tablet 6e46e30a4a424e80b9cfdec12b0875fc (DEFAULT_TABLE table=test-table [id=2ed61e6bb8af4cd88cb79a01e49bdabf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:09.606932 22969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e46e30a4a424e80b9cfdec12b0875fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:09.610240 23367 ts_tablet_manager.cc:1918] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:09.610361 23367 log.cc:1196] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/c7cfd0f3042042aea47aca65f5cc0c11
I20260924 03:16:09.610769 23367 ts_tablet_manager.cc:1939] T c7cfd0f3042042aea47aca65f5cc0c11 P 3943d37a1703465093f902bd7289e278: Deleting consensus metadata
I20260924 03:16:09.611760 22818 catalog_manager.cc:4841] TS 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783): tablet c7cfd0f3042042aea47aca65f5cc0c11 (table test-table [id=a04b55abb13342ad83ffb8be4123596e]) successfully deleted
I20260924 03:16:09.612668 23343 tablet_bootstrap.cc:492] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: Bootstrap starting.
I20260924 03:16:09.612713 23345 tablet_bootstrap.cc:492] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: Bootstrap starting.
I20260924 03:16:09.614240 23345 tablet_bootstrap.cc:654] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:09.614377 23343 tablet_bootstrap.cc:654] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:09.614787 23369 ts_tablet_manager.cc:1918] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:09.614902 23369 log.cc:1196] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/c7cfd0f3042042aea47aca65f5cc0c11
I20260924 03:16:09.615397 23369 ts_tablet_manager.cc:1939] T c7cfd0f3042042aea47aca65f5cc0c11 P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting consensus metadata
I20260924 03:16:09.616487 23342 tablet_bootstrap.cc:492] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: Bootstrap starting.
I20260924 03:16:09.616541 22820 catalog_manager.cc:4841] TS cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981): tablet c7cfd0f3042042aea47aca65f5cc0c11 (table test-table [id=a04b55abb13342ad83ffb8be4123596e]) successfully deleted
I20260924 03:16:09.618404 23342 tablet_bootstrap.cc:654] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:09.623431 23343 tablet_bootstrap.cc:492] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: No bootstrap required, opened a new log
I20260924 03:16:09.623665 23343 ts_tablet_manager.cc:1397] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 03:16:09.624511 23343 raft_consensus.cc:348] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.624881 23343 raft_consensus.cc:374] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:09.625000 23343 raft_consensus.cc:729] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943d37a1703465093f902bd7289e278, State: Initialized, Role: FOLLOWER
I20260924 03:16:09.625427 23345 tablet_bootstrap.cc:492] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: No bootstrap required, opened a new log
I20260924 03:16:09.625295 23343 consensus_queue.cc:260] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.625885 23343 ts_tablet_manager.cc:1428] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:16:09.626299 23345 ts_tablet_manager.cc:1397] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260924 03:16:09.627197 23345 raft_consensus.cc:348] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.627430 23345 raft_consensus.cc:374] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:09.627508 23345 raft_consensus.cc:729] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f5bf792d454d54b85ec47d4715dc54, State: Initialized, Role: FOLLOWER
I20260924 03:16:09.627771 23345 consensus_queue.cc:260] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.629068 23345 ts_tablet_manager.cc:1428] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 03:16:09.636822 23342 tablet_bootstrap.cc:492] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: No bootstrap required, opened a new log
I20260924 03:16:09.637063 23342 ts_tablet_manager.cc:1397] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20260924 03:16:09.637918 23342 raft_consensus.cc:348] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.638212 23342 raft_consensus.cc:374] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:09.638314 23342 raft_consensus.cc:729] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8066f0d4e34e36a2cd9d47cb3c73dd, State: Initialized, Role: FOLLOWER
I20260924 03:16:09.638608 23342 consensus_queue.cc:260] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.639129 23342 ts_tablet_manager.cc:1428] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:16:09.757753 23396 raft_consensus.cc:482] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:16:09.758018 23396 raft_consensus.cc:504] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.758749 23396 leader_election.cc:288] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689), cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:09.759178 22990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e46e30a4a424e80b9cfdec12b0875fc" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f5bf792d454d54b85ec47d4715dc54" is_pre_election: true
I20260924 03:16:09.759323 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e46e30a4a424e80b9cfdec12b0875fc" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" is_pre_election: true
I20260924 03:16:09.759449 22990 raft_consensus.cc:2454] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 0.
I20260924 03:16:09.759560 23130 raft_consensus.cc:2454] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 0.
I20260924 03:16:09.759940 23199 leader_election.cc:302] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3943d37a1703465093f902bd7289e278, 64f5bf792d454d54b85ec47d4715dc54; no voters:
I20260924 03:16:09.760239 23396 raft_consensus.cc:2781] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:16:09.760363 23396 raft_consensus.cc:482] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:16:09.760510 23396 raft_consensus.cc:3037] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:09.763724 23396 raft_consensus.cc:504] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.764344 23396 leader_election.cc:288] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 election: Requested vote from peers 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689), cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981)
I20260924 03:16:09.764806 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e46e30a4a424e80b9cfdec12b0875fc" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd"
I20260924 03:16:09.764842 22990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e46e30a4a424e80b9cfdec12b0875fc" candidate_uuid: "3943d37a1703465093f902bd7289e278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f5bf792d454d54b85ec47d4715dc54"
I20260924 03:16:09.765018 23130 raft_consensus.cc:3037] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:09.765062 22990 raft_consensus.cc:3037] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:09.767136 22990 raft_consensus.cc:2454] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 1.
I20260924 03:16:09.767531 23199 leader_election.cc:302] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3943d37a1703465093f902bd7289e278, 64f5bf792d454d54b85ec47d4715dc54; no voters:
I20260924 03:16:09.767809 23396 raft_consensus.cc:2781] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:09.767979 23396 raft_consensus.cc:686] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 1 LEADER]: Becoming Leader. State: Replica: 3943d37a1703465093f902bd7289e278, State: Running, Role: LEADER
I20260924 03:16:09.767997 23130 raft_consensus.cc:2454] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943d37a1703465093f902bd7289e278 in term 1.
I20260924 03:16:09.768251 23396 consensus_queue.cc:237] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } }
I20260924 03:16:09.771095 22832 catalog_manager.cc:5495] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3943d37a1703465093f902bd7289e278 (127.14.227.131). New cstate: current_term: 1 leader_uuid: "3943d37a1703465093f902bd7289e278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3943d37a1703465093f902bd7289e278" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32783 } health_report { overall_health: HEALTHY } } }
I20260924 03:16:09.800176 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.794445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:58337 (local address 127.14.227.131:32783)
0924 03:16:09.794595 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:09.794600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:09.795026 (+ 426us) server_negotiation.cc:407] Connection header received
0924 03:16:09.795225 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:09.795229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:09.795349 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:09.795479 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:09.795980 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.796919 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.798406 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.798815 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.798952 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.799319 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:09.799418 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:09.799514 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:09.799652 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 03:16:09.799723 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 03:16:09.800285 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.794476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:32783 (local address 127.14.227.131:58337)
0924 03:16:09.794932 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:09.794946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:09.795115 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:09.795499 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:09.795504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:09.795514 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 03:16:09.795818 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:09.795826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.797042 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.797048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:09.798285 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:09.798296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.798993 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.798999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:09.799146 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.799177 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 03:16:09.799512 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 03:16:09.799522 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:16:09.799618 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:16:09.799761 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":1,"thread_start_us":241,"threads_started":1}
I20260924 03:16:09.808560 22990 raft_consensus.cc:1261] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Refusing update from remote peer 3943d37a1703465093f902bd7289e278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:09.808686 23130 raft_consensus.cc:1261] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Refusing update from remote peer 3943d37a1703465093f902bd7289e278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:09.809134 23349 consensus_queue.cc:1035] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f5bf792d454d54b85ec47d4715dc54" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:09.809626 23422 consensus_queue.cc:1035] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8066f0d4e34e36a2cd9d47cb3c73dd" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:09.860754 22832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:09.860985 22832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:09.863718 22832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbe754dc06da4684b4935e59ff045c23: Sending DeleteTablet for 3 replicas of tablet 6e46e30a4a424e80b9cfdec12b0875fc
I20260924 03:16:09.864382 22969 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e46e30a4a424e80b9cfdec12b0875fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56266
I20260924 03:16:09.864580 23108 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e46e30a4a424e80b9cfdec12b0875fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:53000
I20260924 03:16:09.864686 23368 tablet_replica.cc:330] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: stopping tablet replica
I20260924 03:16:09.864853 23369 tablet_replica.cc:330] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: stopping tablet replica
I20260924 03:16:09.864887 23368 raft_consensus.cc:2229] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:09.865063 23368 raft_consensus.cc:2258] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.865060 23369 raft_consensus.cc:2229] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:09.865224 23369 raft_consensus.cc:2258] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.865867 23248 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e46e30a4a424e80b9cfdec12b0875fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:09 UTC) from {username='oryx', principal='oryx/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:58062
I20260924 03:16:09.866206 23367 tablet_replica.cc:330] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: stopping tablet replica
I20260924 03:16:09.866206 23368 ts_tablet_manager.cc:1905] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:09.866413 23367 raft_consensus.cc:2229] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:16:09.866719 23367 raft_consensus.cc:2258] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:09.868070 23367 ts_tablet_manager.cc:1905] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:09.868175 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 22904
I20260924 03:16:09.876752 23367 ts_tablet_manager.cc:1918] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:16:09.876931 23367 log.cc:1196] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/6e46e30a4a424e80b9cfdec12b0875fc
I20260924 03:16:09.877489 23367 ts_tablet_manager.cc:1939] T 6e46e30a4a424e80b9cfdec12b0875fc P 3943d37a1703465093f902bd7289e278: Deleting consensus metadata
I20260924 03:16:09.878708 23368 ts_tablet_manager.cc:1918] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:16:09.878865 23368 log.cc:1196] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/6e46e30a4a424e80b9cfdec12b0875fc
I20260924 03:16:09.878808 22818 catalog_manager.cc:4841] TS 3943d37a1703465093f902bd7289e278 (127.14.227.131:32783): tablet 6e46e30a4a424e80b9cfdec12b0875fc (table test-table [id=2ed61e6bb8af4cd88cb79a01e49bdabf]) successfully deleted
I20260924 03:16:09.879261 23368 ts_tablet_manager.cc:1939] T 6e46e30a4a424e80b9cfdec12b0875fc P 64f5bf792d454d54b85ec47d4715dc54: Deleting consensus metadata
I20260924 03:16:09.881212 23369 ts_tablet_manager.cc:1905] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:09.885293 22817 catalog_manager.cc:4841] TS 64f5bf792d454d54b85ec47d4715dc54 (127.14.227.129:40689): tablet 6e46e30a4a424e80b9cfdec12b0875fc (table test-table [id=2ed61e6bb8af4cd88cb79a01e49bdabf]) successfully deleted
I20260924 03:16:09.885656 23369 ts_tablet_manager.cc:1918] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 03:16:09.885794 23369 log.cc:1196] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/6e46e30a4a424e80b9cfdec12b0875fc
I20260924 03:16:09.886235 23369 ts_tablet_manager.cc:1939] T 6e46e30a4a424e80b9cfdec12b0875fc P cd8066f0d4e34e36a2cd9d47cb3c73dd: Deleting consensus metadata
I20260924 03:16:09.887037 22820 catalog_manager.cc:4841] TS cd8066f0d4e34e36a2cd9d47cb3c73dd (127.14.227.130:32981): tablet 6e46e30a4a424e80b9cfdec12b0875fc (table test-table [id=2ed61e6bb8af4cd88cb79a01e49bdabf]) successfully deleted
Sep 24 03:16:09 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.129@KRBTEST.COM
I20260924 03:16:09.907724 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:09.868986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.14.227.129:40689)
0924 03:16:09.885613 (+ 16627us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:09.885619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:09.885638 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:16:09.885694 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:09.885699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:09.885787 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:09.885921 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:09.887611 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.888284 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.890089 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:09.890471 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:09.890559 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:09.900787 (+ 10228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:09.900807 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:09.900829 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:09.900882 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:09.904696 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:09.905371 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:09.905382 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:09.905385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:09.905552 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:09.905918 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:09.905923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:09.905927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:09.906430 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:09.906758 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:09.906997 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 03:16:09.907258 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16150,"thread_start_us":16046,"threads_started":1}
I20260924 03:16:09.908460 23032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:10.033275 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 22904
I20260924 03:16:10.055146 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 23044
Sep 24 03:16:10 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.130@KRBTEST.COM
I20260924 03:16:10.071291 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:10.055876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.14.227.130:32981)
0924 03:16:10.056401 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:10.056405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:10.056419 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:16:10.056475 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:10.056478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:10.056537 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:10.056652 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:10.057815 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:10.058636 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:10.059973 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:10.060435 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:10.060562 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:10.065009 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:10.065024 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:10.065039 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:10.065074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:10.068350 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:10.069003 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:10.069012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:10.069014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:10.069136 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:10.069571 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:10.069577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:10.069580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:10.069919 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:10.070327 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:10.070669 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 03:16:10.070920 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":283,"threads_started":1}
I20260924 03:16:10.071980 23174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:10.184334 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 23044
I20260924 03:16:10.202572 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 23183
Sep 24 03:16:10 dist-test-slave-vd0p krb5kdc[22788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.227.131@KRBTEST.COM
I20260924 03:16:10.217082 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:10.203364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.14.227.131:32783)
0924 03:16:10.203569 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:10.203573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:10.203594 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:16:10.203747 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:10.203751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:10.203835 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:10.203928 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:10.205165 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:10.205871 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:10.207245 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:10.207668 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:10.207853 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:10.212010 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:10.212025 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:10.212040 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:10.212079 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:10.214816 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:10.215410 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:10.215416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:10.215418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:10.215547 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:10.215850 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:10.215853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:10.215855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:10.216169 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:10.216384 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:10.216641 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 03:16:10.216810 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:16:10.217733 23313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:10.360596 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 23183
I20260924 03:16:10.385675 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 22801
I20260924 03:16:10.387111 22873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:10.500906 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 22801
2026-09-24T03:16:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5749 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:16:10 dist-test-slave-vd0p krb5kdc[23444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:16:10 dist-test-slave-vd0p krb5kdc[23444](info): set up 2 sockets
Sep 24 03:16:10 dist-test-slave-vd0p krb5kdc[23444](info): commencing operation
krb5kdc: starting...
W20260924 03:16:11.602864 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:16:11 dist-test-slave-vd0p krb5kdc[23444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:16:11.811496 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46359
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:46359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:12.114684 23457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:12.114976 23457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:12.115145 23457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:12.122105 23457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:16:12.122197 23457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:12.122301 23457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:12.122375 23457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:16:12.122437 23457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:16:12.131772 23457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:46359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:12.133004 23457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:12.134315 23457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:12.145419 23463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:12.145480 23462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:12.145480 23465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:12.146492 23457 server_base.cc:1034] running on GCE node
I20260924 03:16:12.147212 23457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:12.148478 23457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:12.149704 23457 hybrid_clock.cc:648] HybridClock initialized: now 1790219772149669 us; error 43 us; skew 500 ppm
I20260924 03:16:12.150077 23457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T03:16:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1714 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:16:12 dist-test-slave-vd0p krb5kdc[23471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:16:12 dist-test-slave-vd0p krb5kdc[23471](info): set up 2 sockets
Sep 24 03:16:12 dist-test-slave-vd0p krb5kdc[23471](info): commencing operation
krb5kdc: starting...
W20260924 03:16:13.290853 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:16:13 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:16:13.499387 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43591
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:13.813180 23484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:13.813513 23484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:13.813675 23484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:13.820784 23484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:16:13.820924 23484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:13.821035 23484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:13.821134 23484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:16:13.821208 23484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:16:13.830878 23484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:13.832162 23484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:13.833455 23484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:13.845335 23489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:13.845394 23490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:13.845314 23492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:13.846339 23484 server_base.cc:1034] running on GCE node
I20260924 03:16:13.847083 23484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:13.848380 23484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:13.849603 23484 hybrid_clock.cc:648] HybridClock initialized: now 1790219773849568 us; error 51 us; skew 500 ppm
I20260924 03:16:13.850004 23484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:13 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219773, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:13.853082 23484 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:16:13.854519 23484 webserver.cc:460] Webserver started at http://127.14.227.190:43255/ using document root <none> and password file <none>
I20260924 03:16:13.855186 23484 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:13.855329 23484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:13.855634 23484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:13.858551 23484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "9f65c96f67e2447caaf07c8ae5e461b1"
format_stamp: "Formatted at 2026-09-24 03:16:13 on dist-test-slave-vd0p"
server_key: "ec689ab7d7e79920fff939f623acc83a"
server_key_iv: "cd69ced1c29d107ebac24a9a3127d329"
server_key_version: "encryptionkey@0"
I20260924 03:16:13.859280 23484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "9f65c96f67e2447caaf07c8ae5e461b1"
format_stamp: "Formatted at 2026-09-24 03:16:13 on dist-test-slave-vd0p"
server_key: "ec689ab7d7e79920fff939f623acc83a"
server_key_iv: "cd69ced1c29d107ebac24a9a3127d329"
server_key_version: "encryptionkey@0"
I20260924 03:16:13.864329 23484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:16:13.867550 23499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:13.868750 23484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:16:13.868927 23484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "9f65c96f67e2447caaf07c8ae5e461b1"
format_stamp: "Formatted at 2026-09-24 03:16:13 on dist-test-slave-vd0p"
server_key: "ec689ab7d7e79920fff939f623acc83a"
server_key_iv: "cd69ced1c29d107ebac24a9a3127d329"
server_key_version: "encryptionkey@0"
I20260924 03:16:13.869112 23484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:13.898558 23484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:13.902870 23484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:13.903157 23484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:13.920920 23484 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:43591
I20260924 03:16:13.920940 23550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:43591 every 8 connection(s)
I20260924 03:16:13.922344 23484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:16:13.925884 23551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:13.927915 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 23484
I20260924 03:16:13.928149 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:16:13.928932 15246 external_mini_cluster.cc:1428] Setting key c642b09dfdcdb30ad5d313dc0986e210
I20260924 03:16:13.938284 23551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Bootstrap starting.
I20260924 03:16:13.942117 23551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Neither blocks nor log segments found. Creating new log.
Sep 24 03:16:13 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:16:13.943221 23551 log.cc:824] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:13.946326 23551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: No bootstrap required, opened a new log
I20260924 03:16:13.951015 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:13.930955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.14.227.190:43591)
0924 03:16:13.931741 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:13.931752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:13.931814 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:16:13.933419 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:13.933440 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:13.933804 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:13.934166 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:13.935905 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:13.937160 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:13.938823 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:13.939438 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:13.939562 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:13.944159 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:13.944171 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:13.944192 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:13.944238 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:13.947266 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:13.948020 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:13.948028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:13.948033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:13.948207 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:13.948591 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:13.948594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:13.948595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:13.949070 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:13.949269 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:13.949671 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 03:16:13.950051 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":364,"threads_started":1}
I20260924 03:16:13.954958 23551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } }
I20260924 03:16:13.955307 23551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:13.955408 23551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f65c96f67e2447caaf07c8ae5e461b1, State: Initialized, Role: FOLLOWER
I20260924 03:16:13.955930 23551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } }
I20260924 03:16:13.956207 23551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:16:13.956386 23551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:16:13.956609 23551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:13.958825 23551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } }
I20260924 03:16:13.959311 23551 leader_election.cc:302] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f65c96f67e2447caaf07c8ae5e461b1; no voters:
I20260924 03:16:13.959892 23551 leader_election.cc:288] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:16:13.960011 23556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:13.960686 23556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 1 LEADER]: Becoming Leader. State: Replica: 9f65c96f67e2447caaf07c8ae5e461b1, State: Running, Role: LEADER
I20260924 03:16:13.961133 23556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } }
I20260924 03:16:13.961917 23551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:16:13.964313 23558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f65c96f67e2447caaf07c8ae5e461b1. Latest consensus state: current_term: 1 leader_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } } }
I20260924 03:16:13.964751 23557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } } }
I20260924 03:16:13.965426 23558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:13.965466 23557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:13.967027 23565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:16:13.971025 23565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:16:13.979449 23565 catalog_manager.cc:1347] Generated new cluster ID: e5d1d55d6f204d5f8f1b0c7a8d077422
I20260924 03:16:13.979566 23565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:16:13.990469 23565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:16:13.992244 23565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:16:14.005744 23565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Generated new TSK 0
I20260924 03:16:14.006778 23565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:16:14.070245 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43591
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:14.386031 23577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:14.386363 23577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:14.386541 23577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:14.393569 23577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:14.393671 23577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:14.393857 23577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:16:14.403774 23577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:14.405021 23577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:14.406513 23577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:14.419416 23585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:14.419487 23582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:14.419463 23583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:14.420428 23577 server_base.cc:1034] running on GCE node
I20260924 03:16:14.421015 23577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:14.421998 23577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:14.423209 23577 hybrid_clock.cc:648] HybridClock initialized: now 1790219774423140 us; error 56 us; skew 500 ppm
I20260924 03:16:14.423579 23577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:14 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219774, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:14.426789 23577 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:16:14.428156 23577 webserver.cc:460] Webserver started at http://127.14.227.129:34387/ using document root <none> and password file <none>
I20260924 03:16:14.428768 23577 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:14.428871 23577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:14.429111 23577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:14.431833 23577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "a7a064a1e1414eecab3f047f96020559"
format_stamp: "Formatted at 2026-09-24 03:16:14 on dist-test-slave-vd0p"
server_key: "9f2e2924fa48dbfdaddd8d4a5dc1bfb5"
server_key_iv: "9c90557dfd828fef10d5cf2a5836d54b"
server_key_version: "encryptionkey@0"
I20260924 03:16:14.432531 23577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "a7a064a1e1414eecab3f047f96020559"
format_stamp: "Formatted at 2026-09-24 03:16:14 on dist-test-slave-vd0p"
server_key: "9f2e2924fa48dbfdaddd8d4a5dc1bfb5"
server_key_iv: "9c90557dfd828fef10d5cf2a5836d54b"
server_key_version: "encryptionkey@0"
I20260924 03:16:14.437397 23577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:16:14.440779 23592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:14.442159 23577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:16:14.442288 23577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "a7a064a1e1414eecab3f047f96020559"
format_stamp: "Formatted at 2026-09-24 03:16:14 on dist-test-slave-vd0p"
server_key: "9f2e2924fa48dbfdaddd8d4a5dc1bfb5"
server_key_iv: "9c90557dfd828fef10d5cf2a5836d54b"
server_key_version: "encryptionkey@0"
I20260924 03:16:14.442452 23577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:14.462261 23577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:14.466521 23577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:14.466753 23577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:14.467675 23577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:14.468961 23577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:14.469033 23577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:14.469115 23577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:14.469156 23577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:14.502146 23577 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:33975
I20260924 03:16:14.502190 23704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:33975 every 8 connection(s)
I20260924 03:16:14.503430 23577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:16:14.512123 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 23577
I20260924 03:16:14.512342 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:16:14.512964 15246 external_mini_cluster.cc:1428] Setting key b504030ed062f1d787f7a76077eb959f
Sep 24 03:16:14 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219774, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:16:14.524173 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:14.505936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:42565 (local address 127.14.227.190:43591)
0924 03:16:14.506387 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:14.506391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:14.507307 (+ 916us) server_negotiation.cc:407] Connection header received
0924 03:16:14.509098 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:14.509104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:14.509188 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:14.509308 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:14.510798 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:14.511499 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:14.513161 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:14.513711 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:14.513862 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:14.517743 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:14.517751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:14.517758 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:14.517821 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:14.521082 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:14.521810 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:14.521818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:14.521824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:14.522006 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:14.522490 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:14.522493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:14.522495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:14.522760 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:14.522878 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:14.523760 (+ 882us) server_negotiation.cc:299] Negotiation successful
0924 03:16:14.523921 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":208,"threads_started":1}
I20260924 03:16:14.525736 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:14.506500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:42565)
0924 03:16:14.507132 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:14.507186 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:14.508845 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:14.509529 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:14.509537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:14.510089 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:14.510617 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:14.510632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:14.511650 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:14.511655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:14.512998 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:14.513010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:14.513970 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:14.513980 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:14.514219 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:14.514970 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:14.514998 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:14.517539 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:14.521216 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:14.521223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:14.521236 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:14.521667 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:14.522112 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:14.522119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:14.522125 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:14.522380 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:14.522895 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:14.522904 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:14.523359 (+ 455us) client_negotiation.cc:769] Sending connection context
0924 03:16:14.523681 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 03:16:14.524259 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":240,"threads_started":1}
I20260924 03:16:14.527360 23705 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43591
I20260924 03:16:14.527693 23705 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:14.528369 23705 heartbeater.cc:511] Master 127.14.227.190:43591 requested a full tablet report, sending...
I20260924 03:16:14.530359 23516 ts_manager.cc:194] Registered new tserver with Master: a7a064a1e1414eecab3f047f96020559 (127.14.227.129:33975)
I20260924 03:16:14.531924 23516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:42565
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:16:14.560410 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43591
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:14.861393 23711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:14.861738 23711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:14.861902 23711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:14.868908 23711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:14.868996 23711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:14.869096 23711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:16:14.878204 23711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:14.879245 23711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:14.880419 23711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:14.892755 23719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:14.892765 23717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:14.892817 23716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:14.893738 23711 server_base.cc:1034] running on GCE node
I20260924 03:16:14.894362 23711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:14.895253 23711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:14.896428 23711 hybrid_clock.cc:648] HybridClock initialized: now 1790219774896370 us; error 55 us; skew 500 ppm
I20260924 03:16:14.896752 23711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:14 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219774, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:14.903231 23711 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:16:14.904560 23711 webserver.cc:460] Webserver started at http://127.14.227.130:46415/ using document root <none> and password file <none>
I20260924 03:16:14.905207 23711 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:14.905344 23711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:14.905632 23711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:14.908392 23711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "008f62719790476d8cd59cf10aa04944"
format_stamp: "Formatted at 2026-09-24 03:16:14 on dist-test-slave-vd0p"
server_key: "d5a1bef5ba3eea9e68641b6565e90657"
server_key_iv: "b7d1b7477289b8bbe243aca9da7d4dbf"
server_key_version: "encryptionkey@0"
I20260924 03:16:14.909142 23711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "008f62719790476d8cd59cf10aa04944"
format_stamp: "Formatted at 2026-09-24 03:16:14 on dist-test-slave-vd0p"
server_key: "d5a1bef5ba3eea9e68641b6565e90657"
server_key_iv: "b7d1b7477289b8bbe243aca9da7d4dbf"
server_key_version: "encryptionkey@0"
I20260924 03:16:14.913884 23711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 03:16:14.916997 23726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:14.918253 23711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:16:14.918418 23711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "008f62719790476d8cd59cf10aa04944"
format_stamp: "Formatted at 2026-09-24 03:16:14 on dist-test-slave-vd0p"
server_key: "d5a1bef5ba3eea9e68641b6565e90657"
server_key_iv: "b7d1b7477289b8bbe243aca9da7d4dbf"
server_key_version: "encryptionkey@0"
I20260924 03:16:14.918610 23711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:14.940703 23711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:14.944764 23711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:14.945040 23711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:14.946105 23711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:14.947439 23711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:14.947539 23711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:14.947695 23711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:14.947764 23711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:14.979440 23711 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:35697
I20260924 03:16:14.979490 23838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:35697 every 8 connection(s)
I20260924 03:16:14.980670 23711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:16:14.987870 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 23711
I20260924 03:16:14.988034 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:16:14.988487 15246 external_mini_cluster.cc:1428] Setting key ff8b94df9014c0b4424e314f4fc32c7d
Sep 24 03:16:14 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219774, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:16:15.000192 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:14.983074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:52789 (local address 127.14.227.190:43591)
0924 03:16:14.983246 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:14.983250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:14.984278 (+ 1028us) server_negotiation.cc:407] Connection header received
0924 03:16:14.985917 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:14.985922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:14.986039 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:14.986124 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:14.987627 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:14.988442 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:14.990525 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:14.990897 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:14.990988 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:14.994840 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:14.994845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:14.994849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:14.994877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:14.997355 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:14.998068 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:14.998076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:14.998079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:14.998208 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:14.998656 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:14.998664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:14.998667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:14.998902 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:14.999055 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:14.999668 (+ 613us) server_negotiation.cc:299] Negotiation successful
0924 03:16:14.999871 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:16:15.001076 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:14.983463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:52789)
0924 03:16:14.984111 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:14.984147 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:14.985689 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:14.986318 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:14.986325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:14.986834 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:14.987371 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:14.987390 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:14.988584 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:14.988589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:14.990348 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:14.990361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:14.991106 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:14.991112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:14.991406 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:14.992076 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:14.992108 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:14.994631 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:14.997490 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:14.997499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:14.997514 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:14.997912 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:14.998328 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:14.998333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:14.998336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:14.998526 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:14.999023 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:14.999031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:14.999338 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 03:16:14.999571 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 03:16:15.000031 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":259,"threads_started":1}
I20260924 03:16:15.002521 23839 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43591
I20260924 03:16:15.002835 23839 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:15.003664 23839 heartbeater.cc:511] Master 127.14.227.190:43591 requested a full tablet report, sending...
I20260924 03:16:15.005272 23516 ts_manager.cc:194] Registered new tserver with Master: 008f62719790476d8cd59cf10aa04944 (127.14.227.130:35697)
I20260924 03:16:15.006608 23516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:52789
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:16:15.036257 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43591
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:15.340451 23844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:15.340703 23844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:15.340790 23844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:15.347728 23844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:15.347823 23844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:15.347927 23844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:16:15.356956 23844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:15.358096 23844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:15.359311 23844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:15.372035 23852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:15.372081 23850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:15.372131 23849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:15.372769 23844 server_base.cc:1034] running on GCE node
I20260924 03:16:15.373309 23844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:15.374220 23844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:15.375432 23844 hybrid_clock.cc:648] HybridClock initialized: now 1790219775375400 us; error 43 us; skew 500 ppm
I20260924 03:16:15.375721 23844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:15 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219775, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:15.378623 23844 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:16:15.380206 23844 webserver.cc:460] Webserver started at http://127.14.227.131:42067/ using document root <none> and password file <none>
I20260924 03:16:15.381043 23844 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:15.381163 23844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:15.381546 23844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:15.384765 23844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "d7794da4161f43baa4619a4632f82d9d"
format_stamp: "Formatted at 2026-09-24 03:16:15 on dist-test-slave-vd0p"
server_key: "e032dec23a2519349dd5ef54cb3d73ed"
server_key_iv: "576fbdde046937663abc06b69651ed59"
server_key_version: "encryptionkey@0"
I20260924 03:16:15.385536 23844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "d7794da4161f43baa4619a4632f82d9d"
format_stamp: "Formatted at 2026-09-24 03:16:15 on dist-test-slave-vd0p"
server_key: "e032dec23a2519349dd5ef54cb3d73ed"
server_key_iv: "576fbdde046937663abc06b69651ed59"
server_key_version: "encryptionkey@0"
I20260924 03:16:15.392373 23844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260924 03:16:15.395632 23859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:15.396745 23844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:16:15.396893 23844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "d7794da4161f43baa4619a4632f82d9d"
format_stamp: "Formatted at 2026-09-24 03:16:15 on dist-test-slave-vd0p"
server_key: "e032dec23a2519349dd5ef54cb3d73ed"
server_key_iv: "576fbdde046937663abc06b69651ed59"
server_key_version: "encryptionkey@0"
I20260924 03:16:15.397087 23844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:15.410811 23844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:15.414389 23844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:15.414671 23844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:15.415513 23844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:15.416802 23844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:15.416894 23844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:15.417034 23844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:15.417105 23844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:15.448215 23844 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:43379
I20260924 03:16:15.448251 23971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:43379 every 8 connection(s)
I20260924 03:16:15.449469 23844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:16:15.454375 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 23844
I20260924 03:16:15.454567 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:16:15.455129 15246 external_mini_cluster.cc:1428] Setting key ca18f4e8100f331eb7ffc57ee11759c7
Sep 24 03:16:15 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219775, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:16:15.468914 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:15.451866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:34925 (local address 127.14.227.190:43591)
0924 03:16:15.452021 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:15.452025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:15.453079 (+ 1054us) server_negotiation.cc:407] Connection header received
0924 03:16:15.455273 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:15.455277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:15.455342 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:15.455426 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:15.457322 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.458373 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.459859 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.460248 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.460337 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:15.463704 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:15.463709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:15.463712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:15.463743 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:15.466366 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:15.466974 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:15.466978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:15.466980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:15.467086 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:15.467418 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:15.467421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:15.467422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:15.467662 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:15.467771 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:15.468312 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 03:16:15.468492 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:16:15.470145 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:15.452269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:34925)
0924 03:16:15.452920 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:15.452959 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:15.454976 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:15.455702 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:15.455710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:15.456344 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:15.457173 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:15.457193 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.458520 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.458524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:15.459745 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:15.459756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.460413 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.460417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:15.460602 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:15.461245 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:15.461284 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:15.463526 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:15.466483 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:15.466489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:15.466499 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:15.466884 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:15.467179 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:15.467182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:15.467184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:15.467336 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:15.467765 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:15.467770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:15.468030 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 03:16:15.468247 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 03:16:15.468781 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":250,"threads_started":1}
I20260924 03:16:15.471349 23972 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43591
I20260924 03:16:15.471640 23972 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:15.472357 23972 heartbeater.cc:511] Master 127.14.227.190:43591 requested a full tablet report, sending...
I20260924 03:16:15.473678 23516 ts_manager.cc:194] Registered new tserver with Master: d7794da4161f43baa4619a4632f82d9d (127.14.227.131:43379)
I20260924 03:16:15.474485 23516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:34925
I20260924 03:16:15.482506 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab.
Entry for principal oryx/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab.
I20260924 03:16:15.534879 23705 heartbeater.cc:503] Master 127.14.227.190:43591 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab.
Entry for principal oryx/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab.
WARNING: no policy specified for oryx/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab.
Entry for principal oryx/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab.
I20260924 03:16:15.607964 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 23577
Sep 24 03:16:15 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:16:15.622594 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:15.609066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.14.227.129:33975)
0924 03:16:15.609729 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:15.609737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:15.609762 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:16:15.609833 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:15.609839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:15.610094 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:15.610237 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:15.611438 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.612368 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.613438 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.614027 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.614127 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:15.617824 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:15.617833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:15.617857 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:15.617883 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:15.620343 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:15.620893 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:15.620899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:15.620901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:15.621004 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:15.621315 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:15.621318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:15.621319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:15.621600 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:15.621801 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:15.622082 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 03:16:15.622314 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":355,"threads_started":1}
I20260924 03:16:15.623822 23700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:15.704394 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 23577
I20260924 03:16:15.719390 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 23711
Sep 24 03:16:15 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:16:15.734441 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:15.720247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.14.227.130:35697)
0924 03:16:15.720800 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:15.720806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:15.720829 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:16:15.720886 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:15.720892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:15.721098 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:15.721259 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:15.722443 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.723520 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.724685 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.725226 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.725313 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:15.729038 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:15.729048 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:15.729060 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:15.729104 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:15.731949 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:15.732618 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:15.732624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:15.732627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:15.732740 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:15.733027 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:15.733030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:15.733032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:15.733329 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:15.733541 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:15.733742 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 03:16:15.733919 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":291,"threads_started":1}
I20260924 03:16:15.735638 23834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:15.815451 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 23711
I20260924 03:16:15.830917 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 23844
Sep 24 03:16:15 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:16:15.846642 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:15.831874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.14.227.131:43379)
0924 03:16:15.832420 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:15.832427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:15.832453 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:16:15.832524 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:15.832530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:15.832732 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:15.832891 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:15.834296 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.835299 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.836504 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:15.837032 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:15.837165 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:15.841572 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:15.841582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:15.841595 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:15.841624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:15.844255 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:15.844826 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:15.844833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:15.844835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:15.844944 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:15.845234 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:15.845237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:15.845239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:15.845535 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:15.845779 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:15.846096 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 03:16:15.846323 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":275,"threads_started":1}
I20260924 03:16:15.847860 23967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:15.929152 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 23844
I20260924 03:16:15.945619 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 23484
I20260924 03:16:15.946823 23546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:16.048293 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 23484
I20260924 03:16:16.068006 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43591
--webserver_interface=127.14.227.190
--webserver_port=43255
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:16.372495 23989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:16.372764 23989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:16.372848 23989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:16.379518 23989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:16:16.379604 23989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:16.379647 23989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:16.379702 23989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:16:16.379743 23989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:16:16.389741 23989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=43255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:16.390872 23989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:16.392076 23989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:16.403174 23997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:16.403260 23994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:16.403225 23995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:16.403996 23989 server_base.cc:1034] running on GCE node
I20260924 03:16:16.404677 23989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:16.405869 23989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:16.407058 23989 hybrid_clock.cc:648] HybridClock initialized: now 1790219776407027 us; error 35 us; skew 500 ppm
I20260924 03:16:16.407361 23989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:16 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219776, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:16.410241 23989 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:16:16.411564 23989 webserver.cc:460] Webserver started at http://127.14.227.190:43255/ using document root <none> and password file <none>
I20260924 03:16:16.412155 23989 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:16.412290 23989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:16.417332 23989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:16:16.419745 24004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:16.420936 23989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:16:16.421065 23989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "9f65c96f67e2447caaf07c8ae5e461b1"
format_stamp: "Formatted at 2026-09-24 03:16:13 on dist-test-slave-vd0p"
server_key: "ec689ab7d7e79920fff939f623acc83a"
server_key_iv: "cd69ced1c29d107ebac24a9a3127d329"
server_key_version: "encryptionkey@0"
I20260924 03:16:16.421933 23989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:16.452980 23989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:16.456593 23989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:16.456818 23989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:16.474280 23989 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:43591
I20260924 03:16:16.474288 24055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:43591 every 8 connection(s)
I20260924 03:16:16.475764 23989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:16:16.476514 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 23989
I20260924 03:16:16.477417 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:33975
--local_ip_for_outbound_sockets=127.14.227.129
--tserver_master_addrs=127.14.227.190:43591
--webserver_port=34387
--webserver_interface=127.14.227.129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.227.129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:16:16.482590 24056 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:16:16.485620 24056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Bootstrap starting.
I20260924 03:16:16.503859 24056 log.cc:824] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:16.512025 24056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:16:16.512581 24056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Bootstrap complete.
I20260924 03:16:16.521274 24056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } }
I20260924 03:16:16.522136 24056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f65c96f67e2447caaf07c8ae5e461b1, State: Initialized, Role: FOLLOWER
I20260924 03:16:16.522684 24056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } }
I20260924 03:16:16.522933 24056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:16:16.523067 24056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:16:16.523252 24056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:16:16.527155 24056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } }
I20260924 03:16:16.527597 24056 leader_election.cc:302] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f65c96f67e2447caaf07c8ae5e461b1; no voters:
I20260924 03:16:16.528162 24056 leader_election.cc:288] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:16:16.528332 24060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:16:16.529383 24060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [term 2 LEADER]: Becoming Leader. State: Replica: 9f65c96f67e2447caaf07c8ae5e461b1, State: Running, Role: LEADER
I20260924 03:16:16.530084 24056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:16:16.530040 24060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } }
I20260924 03:16:16.532670 24061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } } }
I20260924 03:16:16.532490 24062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f65c96f67e2447caaf07c8ae5e461b1. Latest consensus state: current_term: 2 leader_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f65c96f67e2447caaf07c8ae5e461b1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43591 } } }
I20260924 03:16:16.533034 24061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:16.533068 24062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:16.534006 24067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:16:16.547034 24067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:16:16.548858 24067 catalog_manager.cc:1259] Loaded cluster ID: e5d1d55d6f204d5f8f1b0c7a8d077422
I20260924 03:16:16.549048 24067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:16:16.554019 24067 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:16:16.555779 24067 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f65c96f67e2447caaf07c8ae5e461b1: Loaded TSK: 0
I20260924 03:16:16.557578 24067 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:16:16.828871 24058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:16.829180 24058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:16.829347 24058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:16.836581 24058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:16.836726 24058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:16.836905 24058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:16:16.846614 24058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.129.keytab
--principal=oryx/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:33975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=34387
--tserver_master_addrs=127.14.227.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:16.847851 24058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:16.849067 24058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:16.862787 24085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:16.863363 24083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:16.863541 24082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:16.865588 24058 server_base.cc:1034] running on GCE node
I20260924 03:16:16.866215 24058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:16.867128 24058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:16.868381 24058 hybrid_clock.cc:648] HybridClock initialized: now 1790219776868339 us; error 32 us; skew 500 ppm
I20260924 03:16:16.868695 24058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:16 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219776, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:16.871773 24058 init.cc:377] Logged in from keytab as oryx/127.14.227.129@KRBTEST.COM (short username oryx)
I20260924 03:16:16.873207 24058 webserver.cc:460] Webserver started at http://127.14.227.129:34387/ using document root <none> and password file <none>
I20260924 03:16:16.873883 24058 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:16.874069 24058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:16.878968 24058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:16:16.881448 24092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:16.882814 24058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:16:16.882982 24058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "a7a064a1e1414eecab3f047f96020559"
format_stamp: "Formatted at 2026-09-24 03:16:14 on dist-test-slave-vd0p"
server_key: "9f2e2924fa48dbfdaddd8d4a5dc1bfb5"
server_key_iv: "9c90557dfd828fef10d5cf2a5836d54b"
server_key_version: "encryptionkey@0"
I20260924 03:16:16.883874 24058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:16.900960 24058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:16.905256 24058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:16.905534 24058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:16.906607 24058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:16.908000 24058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:16.908087 24058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:16.908226 24058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:16.908319 24058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:16.940927 24058 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:33975
I20260924 03:16:16.941028 24204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:33975 every 8 connection(s)
I20260924 03:16:16.942533 24058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:16:16.947731 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 24058
I20260924 03:16:16.948403 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:35697
--local_ip_for_outbound_sockets=127.14.227.130
--tserver_master_addrs=127.14.227.190:43591
--webserver_port=46415
--webserver_interface=127.14.227.130
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.227.130
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:16:16 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:16 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:16:16.968361 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:16.945326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:60699 (local address 127.14.227.190:43591)
0924 03:16:16.946063 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:16.946075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:16.946402 (+ 327us) server_negotiation.cc:407] Connection header received
0924 03:16:16.948549 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:16.948570 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:16.948981 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:16.949347 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:16.950721 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.952585 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.954088 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.955038 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.955166 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:16.967451 (+ 12285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:60699: BlockingRecv error: recv got EOF from 127.14.227.129:60699 (error 108)
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":260,"threads_started":1}
I20260924 03:16:16.968442 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:16.945450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:60699)
0924 03:16:16.946251 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:16.946295 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:16.947973 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:16.949451 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:16.949458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:16.950007 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:16.950527 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:16.950543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.952734 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.952737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:16.953897 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:16.953918 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.955276 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.955283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:16.955568 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:16.962344 (+ 6776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:16.962378 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:16.967101 (+ 4723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":347,"threads_started":1}
W20260924 03:16:16.969096 24205 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:43591 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.227.190:43591: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Sep 24 03:16:16 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:16 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:16.977674 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:16.969651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:40681)
0924 03:16:16.969865 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:16.969889 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:16.970032 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:16.970328 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:16.970336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:16.970757 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:16.970931 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:16.970941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.971850 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.971855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:16.973284 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:16.973298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.973925 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.973931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:16.974127 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:16.974638 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:16.974657 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:16.977393 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 03:16:16.977783 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:16.969863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:40681 (local address 127.14.227.190:43591)
0924 03:16:16.970032 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:16.970036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:16.970051 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:16:16.970144 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:16.970147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:16.970197 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:16.970267 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:16.971051 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.971728 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.973421 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.973784 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.973877 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:16.977583 (+ 3706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:40681: BlockingRecv error: recv got EOF from 127.14.227.129:40681 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 03:16:16 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:16 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:16.985183 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:16.978289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:41529)
0924 03:16:16.978445 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:16.978461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:16.978560 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:16.978844 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:16.978848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:16.979137 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:16.979258 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:16.979265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.980177 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.980182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:16.981079 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:16.981088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.981728 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.981733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:16.981918 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:16.982484 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:16.982498 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:16.984928 (+ 2430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 03:16:16.985327 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:16.978432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:41529 (local address 127.14.227.190:43591)
0924 03:16:16.978601 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:16.978608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:16.978625 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:16:16.978671 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:16.978674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:16.978727 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:16.978831 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:16.979382 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.980046 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.981209 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:16.981591 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:16.981709 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:16.985087 (+ 3378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:41529: BlockingRecv error: recv got EOF from 127.14.227.129:41529 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 03:16:16.985481 24205 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:16:17.279739 24209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:17.279984 24209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:17.280164 24209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:17.286998 24209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:17.287086 24209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:17.287271 24209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:16:17.296140 24209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.130.keytab
--principal=oryx/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:35697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=46415
--tserver_master_addrs=127.14.227.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:17.297255 24209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:17.298530 24209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:17.310806 24214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:17.310840 24215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:17.310882 24217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:17.311689 24209 server_base.cc:1034] running on GCE node
I20260924 03:16:17.312261 24209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:17.313097 24209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:17.314280 24209 hybrid_clock.cc:648] HybridClock initialized: now 1790219777314243 us; error 31 us; skew 500 ppm
I20260924 03:16:17.314589 24209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219777, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:17.317530 24209 init.cc:377] Logged in from keytab as oryx/127.14.227.130@KRBTEST.COM (short username oryx)
I20260924 03:16:17.318961 24209 webserver.cc:460] Webserver started at http://127.14.227.130:46415/ using document root <none> and password file <none>
I20260924 03:16:17.319561 24209 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:17.319664 24209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:17.324714 24209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 03:16:17.327255 24224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:17.328486 24209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:16:17.328647 24209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "008f62719790476d8cd59cf10aa04944"
format_stamp: "Formatted at 2026-09-24 03:16:14 on dist-test-slave-vd0p"
server_key: "d5a1bef5ba3eea9e68641b6565e90657"
server_key_iv: "b7d1b7477289b8bbe243aca9da7d4dbf"
server_key_version: "encryptionkey@0"
I20260924 03:16:17.329519 24209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:17.339622 24209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:17.343294 24209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:17.343564 24209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:17.344436 24209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:17.345563 24209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:17.345691 24209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:17.345816 24209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:17.345909 24209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:17.377935 24209 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:35697
I20260924 03:16:17.378017 24336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:35697 every 8 connection(s)
I20260924 03:16:17.379362 24209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:16:17.389704 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 24209
I20260924 03:16:17.390769 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:43379
--local_ip_for_outbound_sockets=127.14.227.131
--tserver_master_addrs=127.14.227.190:43591
--webserver_port=42067
--webserver_interface=127.14.227.131
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.227.131
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:16:17.397509 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:17.381741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:60977 (local address 127.14.227.190:43591)
0924 03:16:17.382013 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:17.382020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:17.383197 (+ 1177us) server_negotiation.cc:407] Connection header received
0924 03:16:17.384864 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:17.384868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:17.384932 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:17.385014 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:17.386593 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.388531 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.390846 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.391542 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.391664 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.397293 (+ 5629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:60977: BlockingRecv error: recv got EOF from 127.14.227.130:60977 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:16:17.398277 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:17.382377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:60977)
0924 03:16:17.383041 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:17.383084 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:17.384637 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:17.385200 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:17.385220 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:17.385673 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:17.386371 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.386398 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.388716 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.388722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.390665 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.390683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.391817 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.391827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.392184 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.393074 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:17.393104 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:17.397137 (+ 4033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":247,"threads_started":1}
W20260924 03:16:17.398860 24337 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:43591 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.227.190:43591: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:17.406764 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:17.399666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:60371)
0924 03:16:17.399804 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:17.399820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:17.399921 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:17.400274 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:17.400278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:17.400577 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:17.400702 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.400708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.401710 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.401716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.402629 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.402639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.403319 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.403324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.403484 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.404055 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:17.404071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:17.406517 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 03:16:17.407243 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:17.399846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:60371 (local address 127.14.227.190:43591)
0924 03:16:17.400032 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:17.400037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:17.400052 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:16:17.400098 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:17.400101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:17.400150 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:17.400245 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:17.400849 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.401582 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.402770 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.403154 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.403307 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.406693 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:60371: BlockingRecv error: recv got EOF from 127.14.227.130:60371 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:17.414453 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:17.407610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:58633)
0924 03:16:17.407746 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:17.407769 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:17.407880 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:17.408132 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:17.408136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:17.408417 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:17.408549 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.408555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.409447 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.409452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.410328 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.410338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.410971 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.410977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.411138 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.411656 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:17.411671 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:17.414163 (+ 2492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 03:16:17.414556 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:17.407440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:58633 (local address 127.14.227.190:43591)
0924 03:16:17.407585 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:17.407590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:17.407840 (+ 250us) server_negotiation.cc:407] Connection header received
0924 03:16:17.407964 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:17.407969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:17.408027 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:17.408113 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:17.408669 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.409323 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.410450 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.410842 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.410963 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.414342 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:58633: BlockingRecv error: recv got EOF from 127.14.227.130:58633 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 03:16:17.414738 24337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:16:17.705937 24340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:17.706211 24340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:17.706373 24340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:17.713387 24340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:17.713485 24340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:17.713670 24340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:16:17.723119 24340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/krb5kdc/oryx_127.14.227.131.keytab
--principal=oryx/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:43379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=42067
--tserver_master_addrs=127.14.227.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:17.724372 24340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:17.725709 24340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:17.739012 24345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:17.738950 24346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:17.739358 24348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:17.739651 24340 server_base.cc:1034] running on GCE node
I20260924 03:16:17.740173 24340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:17.741037 24340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:17.742235 24340 hybrid_clock.cc:648] HybridClock initialized: now 1790219777742176 us; error 51 us; skew 500 ppm
I20260924 03:16:17.742542 24340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219777, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:17.745682 24340 init.cc:377] Logged in from keytab as oryx/127.14.227.131@KRBTEST.COM (short username oryx)
I20260924 03:16:17.747022 24340 webserver.cc:460] Webserver started at http://127.14.227.131:42067/ using document root <none> and password file <none>
I20260924 03:16:17.747663 24340 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:17.747808 24340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:17.752641 24340 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:16:17.755157 24355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:17.756378 24340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:16:17.756537 24340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "d7794da4161f43baa4619a4632f82d9d"
format_stamp: "Formatted at 2026-09-24 03:16:15 on dist-test-slave-vd0p"
server_key: "e032dec23a2519349dd5ef54cb3d73ed"
server_key_iv: "576fbdde046937663abc06b69651ed59"
server_key_version: "encryptionkey@0"
I20260924 03:16:17.757413 24340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:17.779479 24340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:17.783655 24340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:17.783941 24340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:17.784891 24340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:17.786212 24340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:17.786334 24340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:17.786456 24340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:17.786605 24340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:17.817868 24340 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:43379
I20260924 03:16:17.817951 24467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:43379 every 8 connection(s)
I20260924 03:16:17.819311 24340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:16:17.821914 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 24340
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 03:16:17.836053 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:17.821615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:34327 (local address 127.14.227.190:43591)
0924 03:16:17.821786 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:17.821790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:17.823338 (+ 1548us) server_negotiation.cc:407] Connection header received
0924 03:16:17.825528 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:17.825532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:17.825613 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:17.825723 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:17.827581 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.828301 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.830106 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.830621 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.830743 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.835793 (+ 5050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:34327: BlockingRecv error: recv got EOF from 127.14.227.131:34327 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 03:16:17.836812 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:17.822351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:34327)
0924 03:16:17.823180 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:17.823232 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:17.825304 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:17.826031 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:17.826047 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:17.826724 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:17.827428 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.827452 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.828437 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.828442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.829982 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.829997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.830842 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.830851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.831050 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.831654 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:17.831677 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:17.835609 (+ 3932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":343,"threads_started":1}
W20260924 03:16:17.837409 24468 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:43591 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.227.190:43591: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
I20260924 03:16:17.839017 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:17.822924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.14.227.190:43591)
0924 03:16:17.823882 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:17.823887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:17.824313 (+ 426us) server_negotiation.cc:407] Connection header received
0924 03:16:17.824451 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:17.824457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:17.824531 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:17.824640 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:17.826240 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.827339 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.828534 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.829097 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.829208 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.832107 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:17.832116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:17.832132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:17.832169 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:17.835684 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:17.836405 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:17.836413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:17.836421 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:17.836617 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:17.837075 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:17.837080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:17.837083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:17.837545 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:17.837877 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:17.838281 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 03:16:17.838611 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":738,"threads_started":1}
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:17.845502 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:17.837931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:48739)
0924 03:16:17.838198 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:17.838215 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:17.838328 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:17.838634 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:17.838638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:17.838999 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:17.839180 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.839200 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.840216 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.840220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.841260 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.841270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.842071 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.842077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.842284 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.842822 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:17.842845 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:17.845255 (+ 2410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160}
W20260924 03:16:17.845613 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:17.838096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:48739 (local address 127.14.227.190:43591)
0924 03:16:17.838246 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:17.838265 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:17.838280 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:16:17.838449 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:17.838454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:17.838515 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:17.838603 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:17.839325 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.840078 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.841387 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.841883 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.842035 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.845407 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:48739: BlockingRecv error: recv got EOF from 127.14.227.131:48739 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:17.852231 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:17.846195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:48003)
0924 03:16:17.846317 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:17.846330 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:17.846395 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:17.846681 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:17.846684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:17.846923 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:17.847043 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.847049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.847878 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.847882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.848691 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.848700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.849238 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.849241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.849346 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.849813 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:17.849827 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:17.852054 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 03:16:17.852419 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:17.846285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:48003 (local address 127.14.227.190:43591)
0924 03:16:17.846490 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:17.846494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:17.846507 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:16:17.846539 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:17.846543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:17.846585 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:17.846653 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:17.847145 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.847775 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.848791 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.849128 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.849209 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.852201 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:48003: BlockingRecv error: recv got EOF from 127.14.227.131:48003 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260924 03:16:17.852516 24468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:17 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:17.994508 24473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:17.986073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:50857)
0924 03:16:17.986566 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:17.986592 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:17.986698 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:17.987023 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:17.987030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:17.987554 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:17.987750 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.987760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.988844 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.988851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.989799 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:17.989810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.990559 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.990565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:17.990756 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.991386 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:17.991402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:17.994199 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":260,"threads_started":1}
W20260924 03:16:17.994727 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:17.986131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:50857 (local address 127.14.227.190:43591)
0924 03:16:17.986304 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:17.986309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:17.986659 (+ 350us) server_negotiation.cc:407] Connection header received
0924 03:16:17.986788 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:17.986792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:17.986849 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:17.986960 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:17.987891 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.988691 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.989947 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:17.990386 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:17.990524 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:17.994445 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:50857: BlockingRecv error: recv got EOF from 127.14.227.129:50857 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 03:16:18 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:18 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:18.423121 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:18.415374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:47397)
0924 03:16:18.415892 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:18.415909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:18.416057 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:18.416395 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:18.416399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:18.416855 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:18.417059 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:18.417067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.417981 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.417988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:18.418943 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:18.418953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.419579 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.419583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:18.419703 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:18.420206 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:18.420222 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:18.422904 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
W20260924 03:16:18.423342 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:18.415402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:47397 (local address 127.14.227.190:43591)
0924 03:16:18.415628 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:18.415638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:18.416014 (+ 376us) server_negotiation.cc:407] Connection header received
0924 03:16:18.416162 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:18.416169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:18.416242 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:18.416338 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:18.417169 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.417831 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.419056 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.419433 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.419520 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:18.423085 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:47397: BlockingRecv error: recv got EOF from 127.14.227.130:47397 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 03:16:18 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:18 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:18.860831 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:18.853058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:40383)
0924 03:16:18.853560 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:18.853577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:18.853727 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:18.854090 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:18.854095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:18.854559 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:18.854731 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:18.854740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.855689 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.855696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:18.856607 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:18.856617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.857278 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.857286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:18.857457 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:18.858067 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:18.858084 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:18.860531 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":287,"threads_started":1}
W20260924 03:16:18.861006 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:18.853097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:40383 (local address 127.14.227.190:43591)
0924 03:16:18.853325 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:18.853332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:18.853663 (+ 331us) server_negotiation.cc:407] Connection header received
0924 03:16:18.853847 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:18.853852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:18.853917 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:18.854038 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:18.854870 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.855543 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.856743 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.857112 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.857246 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:18.860767 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:40383: BlockingRecv error: recv got EOF from 127.14.227.131:40383 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 03:16:18 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:19 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:19.003321 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:18.995407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:54919)
0924 03:16:18.995892 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:18.995909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:18.996014 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:18.996283 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:18.996287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:18.996663 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:18.996821 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:18.996829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.997771 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.997774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:18.998669 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:18.998680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.999309 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.999313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:18.999417 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:19.000011 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:19.000027 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:19.003045 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":283,"threads_started":1}
W20260924 03:16:19.003705 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:18.995504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:54919 (local address 127.14.227.190:43591)
0924 03:16:18.995762 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:18.995769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:18.995984 (+ 215us) server_negotiation.cc:407] Connection header received
0924 03:16:18.996105 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:18.996109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:18.996165 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:18.996239 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:18.996946 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.997659 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.998783 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:18.999178 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:18.999274 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:19.003335 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:54919: BlockingRecv error: recv got EOF from 127.14.227.129:54919 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 24 03:16:19 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:19 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:19.432651 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:19.424088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:49027)
0924 03:16:19.424600 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:19.424623 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:19.424733 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:19.425068 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:19.425072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:19.425624 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:19.425798 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:19.425806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:19.427136 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:19.427140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:19.428169 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:19.428181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:19.428799 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:19.428803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:19.428927 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:19.429438 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:19.429455 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:19.432274 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":292,"threads_started":1}
W20260924 03:16:19.432731 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:19.424210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:49027 (local address 127.14.227.190:43591)
0924 03:16:19.424381 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:19.424385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:19.424696 (+ 311us) server_negotiation.cc:407] Connection header received
0924 03:16:19.424836 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:19.424840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:19.424924 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:19.425024 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:19.425921 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:19.427025 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:19.428287 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:19.428680 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:19.428765 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:19.432480 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:49027: BlockingRecv error: recv got EOF from 127.14.227.130:49027 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 03:16:19 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:19 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:19.870244 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:19.861888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:35041)
0924 03:16:19.862530 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:19.862551 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:19.862687 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:19.863077 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:19.863082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:19.863680 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:19.863922 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:19.863933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:19.864912 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:19.864916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:19.865758 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:19.865767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:19.866490 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:19.866494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:19.866601 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:19.867232 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:19.867249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:19.869872 (+ 2623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":374,"threads_started":1}
W20260924 03:16:19.870324 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:19.862076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:35041 (local address 127.14.227.190:43591)
0924 03:16:19.862271 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:19.862277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:19.862634 (+ 357us) server_negotiation.cc:407] Connection header received
0924 03:16:19.862818 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:19.862823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:19.862898 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:19.863002 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:19.864040 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:19.864773 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:19.865872 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:19.866365 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:19.866457 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:19.870075 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:35041: BlockingRecv error: recv got EOF from 127.14.227.131:35041 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 03:16:20 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:20 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:20.020058 24479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:20.004303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:55307)
0924 03:16:20.004847 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:20.004864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:20.005025 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:20.005593 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:20.005598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:20.006134 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:20.006366 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:20.006380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.007491 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.007503 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:20.008438 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:20.008449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.009261 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.009266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:20.009447 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:20.016482 (+ 7035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:20.016502 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:20.019810 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":283,"threads_started":1}
W20260924 03:16:20.020273 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:20.004441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:55307 (local address 127.14.227.190:43591)
0924 03:16:20.004670 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:20.004677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:20.004952 (+ 275us) server_negotiation.cc:407] Connection header received
0924 03:16:20.005205 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:20.005213 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:20.005292 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:20.005360 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:20.006566 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.007335 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.008663 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.009112 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.009250 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:20.019997 (+ 10747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:55307: BlockingRecv error: recv got EOF from 127.14.227.129:55307 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 03:16:20 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:20 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:20.441778 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:20.433665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:45179)
0924 03:16:20.434191 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:20.434211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:20.434329 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:20.434605 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:20.434608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:20.434956 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:20.435127 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:20.435134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.436139 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.436142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:20.437028 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:20.437038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.437618 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.437621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:20.437744 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:20.438356 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:20.438373 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:20.441484 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":319,"threads_started":1}
W20260924 03:16:20.441943 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:20.433730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:45179 (local address 127.14.227.190:43591)
0924 03:16:20.433899 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:20.433903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:20.434295 (+ 392us) server_negotiation.cc:407] Connection header received
0924 03:16:20.434424 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:20.434427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:20.434487 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:20.434565 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:20.435254 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.436035 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.437138 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.437510 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.437601 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:20.441704 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:45179: BlockingRecv error: recv got EOF from 127.14.227.130:45179 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 03:16:20 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:20 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:20.879818 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:20.871173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:53525)
0924 03:16:20.871670 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:20.871686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:20.871790 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:20.872060 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:20.872063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:20.872394 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:20.872551 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:20.872559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.874001 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.874005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:20.874896 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:20.874907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.875576 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.875582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:20.875763 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:20.876449 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:20.876472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:20.879576 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":281,"threads_started":1}
W20260924 03:16:20.879906 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:20.871273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:53525 (local address 127.14.227.190:43591)
0924 03:16:20.871442 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:20.871447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:20.871753 (+ 306us) server_negotiation.cc:407] Connection header received
0924 03:16:20.871876 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:20.871879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:20.871946 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:20.872019 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:20.872662 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.873831 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.875030 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:20.875434 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:20.875562 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:20.879744 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:53525: BlockingRecv error: recv got EOF from 127.14.227.131:53525 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 03:16:21 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:21 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:21.028633 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:21.020995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:34541)
0924 03:16:21.021437 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:21.021452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:21.021561 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:21.021818 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:21.021822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:21.022231 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:21.022389 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:21.022397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.023331 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.023335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:21.024278 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:21.024289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.024891 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.024895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:21.025010 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:21.025673 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:21.025689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:21.028314 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
W20260924 03:16:21.028877 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:21.021084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:34541 (local address 127.14.227.190:43591)
0924 03:16:21.021345 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:21.021352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:21.021528 (+ 176us) server_negotiation.cc:407] Connection header received
0924 03:16:21.021641 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:21.021644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:21.021701 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:21.021782 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:21.022515 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.023211 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.024401 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.024772 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.024881 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:21.028542 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:34541: BlockingRecv error: recv got EOF from 127.14.227.129:34541 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 24 03:16:21 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:21 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:21.450591 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:21.442767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:42857)
0924 03:16:21.443250 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:21.443265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:21.443375 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:21.443647 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:21.443650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:21.444018 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:21.444164 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:21.444171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.445097 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.445101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:21.446114 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:21.446125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.446778 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.446784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:21.446985 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:21.447551 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:21.447568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:21.450252 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":287,"threads_started":1}
W20260924 03:16:21.450675 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:21.442763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:42857 (local address 127.14.227.190:43591)
0924 03:16:21.442990 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:21.442999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:21.443344 (+ 345us) server_negotiation.cc:407] Connection header received
0924 03:16:21.443468 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:21.443471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:21.443532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:21.443601 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:21.444280 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.444990 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.446223 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.446631 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.446762 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:21.450438 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:42857: BlockingRecv error: recv got EOF from 127.14.227.130:42857 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 03:16:21 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:21 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:21.888919 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:21.880844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:53631)
0924 03:16:21.881375 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:21.881391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:21.881499 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:21.881762 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:21.881765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:21.882222 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:21.882389 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:21.882398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.883436 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.883440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:21.884404 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:21.884414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.885032 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.885036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:21.885154 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:21.885676 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:21.885692 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:21.888550 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":307,"threads_started":1}
W20260924 03:16:21.889055 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:21.880865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:53631 (local address 127.14.227.190:43591)
0924 03:16:21.881070 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:21.881074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:21.881465 (+ 391us) server_negotiation.cc:407] Connection header received
0924 03:16:21.881587 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:21.881591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:21.881645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:21.881721 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:21.882500 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.883323 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.884516 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:21.884899 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:21.885004 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:21.888765 (+ 3761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:53631: BlockingRecv error: recv got EOF from 127.14.227.131:53631 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 03:16:22 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:22 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:22.037810 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:22.029560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:47509)
0924 03:16:22.030115 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:22.030130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:22.030234 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:22.030657 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:22.030660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:22.031195 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:22.031439 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:22.031469 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.032537 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.032541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:22.033570 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:22.033581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.034373 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.034377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:22.034491 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:22.035050 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:22.035065 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:22.037583 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":277,"threads_started":1}
W20260924 03:16:22.037938 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:22.029707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:47509 (local address 127.14.227.190:43591)
0924 03:16:22.029914 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:22.029918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:22.030210 (+ 292us) server_negotiation.cc:407] Connection header received
0924 03:16:22.030351 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:22.030377 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:22.030477 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:22.030600 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:22.031640 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.032398 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.033684 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.034252 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.034346 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:22.037746 (+ 3400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:47509: BlockingRecv error: recv got EOF from 127.14.227.129:47509 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 03:16:22 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:22 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:22.459410 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:22.451536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:39221)
0924 03:16:22.452042 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:22.452057 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:22.452160 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:22.452432 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:22.452435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:22.452790 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:22.452954 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:22.452962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.453936 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.453940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:22.454870 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:22.454880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.455764 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.455768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:22.455908 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:22.456551 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:22.456568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:22.459178 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
W20260924 03:16:22.459590 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:22.451792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:39221 (local address 127.14.227.190:43591)
0924 03:16:22.451962 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:22.451966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:22.452127 (+ 161us) server_negotiation.cc:407] Connection header received
0924 03:16:22.452250 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:22.452254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:22.452311 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:22.452392 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:22.453089 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.453811 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.454995 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.455615 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.455741 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:22.459341 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:39221: BlockingRecv error: recv got EOF from 127.14.227.130:39221 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 03:16:22 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:22 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:22.898090 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:22.890158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:53737)
0924 03:16:22.890691 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:22.890709 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:22.890833 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:22.891170 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:22.891176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:22.891648 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:22.891819 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:22.891828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.892828 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.892836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:22.893739 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:22.893749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.894518 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.894524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:22.894702 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:22.895257 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:22.895273 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:22.897793 (+ 2520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":233,"threads_started":1}
W20260924 03:16:22.898228 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:22.890162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:53737 (local address 127.14.227.190:43591)
0924 03:16:22.890351 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:22.890359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:22.890789 (+ 430us) server_negotiation.cc:407] Connection header received
0924 03:16:22.890964 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:22.890968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:22.891024 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:22.891142 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:22.892006 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.892694 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.893871 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:22.894374 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:22.894508 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:22.898031 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:53737: BlockingRecv error: recv got EOF from 127.14.227.131:53737 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 03:16:23 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:23 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:23.047075 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:23.038919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:60539)
0924 03:16:23.039467 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:23.039483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:23.039589 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:23.039859 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:23.039863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:23.040232 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:23.040391 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:23.040399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.041390 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.041393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:23.042659 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:23.042671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.043594 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.043598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:23.043717 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:23.044239 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:23.044255 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:23.046869 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":299,"threads_started":1}
W20260924 03:16:23.047245 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:23.038949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:60539 (local address 127.14.227.190:43591)
0924 03:16:23.039138 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:23.039142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:23.039553 (+ 411us) server_negotiation.cc:407] Connection header received
0924 03:16:23.039677 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:23.039681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:23.039739 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:23.039818 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:23.040523 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.041278 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.042836 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.043471 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.043563 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:23.047029 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:60539: BlockingRecv error: recv got EOF from 127.14.227.129:60539 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 03:16:23 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:23 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:23.468532 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:23.460367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:53687)
0924 03:16:23.460947 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:23.460962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:23.461107 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:23.461494 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:23.461498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:23.461906 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:23.462121 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:23.462129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.463120 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.463126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:23.464012 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:23.464022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.464726 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.464732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:23.464904 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:23.465505 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:23.465523 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:23.468341 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":326,"threads_started":1}
W20260924 03:16:23.468775 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:23.460537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:53687 (local address 127.14.227.190:43591)
0924 03:16:23.460776 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:23.460785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:23.461043 (+ 258us) server_negotiation.cc:407] Connection header received
0924 03:16:23.461266 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:23.461271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:23.461338 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:23.461438 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:23.462246 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.462970 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.464159 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.464576 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.464711 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:23.468540 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:53687: BlockingRecv error: recv got EOF from 127.14.227.130:53687 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 03:16:23 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:23 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:23.907390 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:23.899219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:36033)
0924 03:16:23.899721 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:23.899738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:23.899850 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:23.900149 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:23.900152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:23.900510 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:23.900670 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:23.900679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.901727 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.901730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:23.902628 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:23.902638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.903330 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.903334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:23.903465 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:23.904137 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:23.904154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:23.907191 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":280,"threads_started":1}
W20260924 03:16:23.907601 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:23.899330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:36033 (local address 127.14.227.190:43591)
0924 03:16:23.899627 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:23.899634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:23.899819 (+ 185us) server_negotiation.cc:407] Connection header received
0924 03:16:23.899954 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:23.899958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:23.900021 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:23.900111 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:23.900791 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.901604 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.902750 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:23.903198 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:23.903301 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:23.907357 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:36033: BlockingRecv error: recv got EOF from 127.14.227.131:36033 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 24 03:16:24 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:24 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:24.056277 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:24.047952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:35173)
0924 03:16:24.048508 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:24.048523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:24.048691 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:24.049005 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:24.049010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:24.049445 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:24.049611 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:24.049620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.050658 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.050664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:24.051588 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:24.051599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.052293 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.052299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:24.052464 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:24.053132 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:24.053149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:24.055931 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":251,"threads_started":1}
W20260924 03:16:24.056471 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:24.048051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:35173 (local address 127.14.227.190:43591)
0924 03:16:24.048278 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:24.048287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:24.048623 (+ 336us) server_negotiation.cc:407] Connection header received
0924 03:16:24.048810 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:24.048821 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:24.048879 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:24.048979 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:24.049755 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.050513 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.051724 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.052142 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.052266 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:24.056196 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:35173: BlockingRecv error: recv got EOF from 127.14.227.129:35173 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 03:16:24 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:24 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:24.477385 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:24.469540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:51459)
0924 03:16:24.470192 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:24.470209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:24.470325 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:24.470610 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:24.470613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:24.471004 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:24.471176 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:24.471185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.472117 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.472120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:24.473068 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:24.473078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.473687 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.473691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:24.473798 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:24.474406 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:24.474423 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:24.477132 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":276,"threads_started":1}
W20260924 03:16:24.477662 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:24.469567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:51459 (local address 127.14.227.190:43591)
0924 03:16:24.469818 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:24.469827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:24.470294 (+ 467us) server_negotiation.cc:407] Connection header received
0924 03:16:24.470416 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:24.470419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:24.470492 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:24.470571 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:24.471289 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.472009 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.473188 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.473563 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.473656 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:24.477404 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:51459: BlockingRecv error: recv got EOF from 127.14.227.130:51459 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 24 03:16:24 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:24 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:24.916453 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:24.908370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:49291)
0924 03:16:24.908902 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:24.908919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:24.909026 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:24.909301 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:24.909305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:24.909682 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:24.909851 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:24.909859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.910805 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.910809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:24.911693 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:24.911703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.912363 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.912370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:24.912523 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:24.913215 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:24.913263 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:24.916081 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":229,"threads_started":1}
W20260924 03:16:24.916476 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:24.908411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:49291 (local address 127.14.227.190:43591)
0924 03:16:24.908612 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:24.908645 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:24.908998 (+ 353us) server_negotiation.cc:407] Connection header received
0924 03:16:24.909120 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:24.909125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:24.909182 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:24.909260 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:24.909997 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.910692 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.911813 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:24.912223 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:24.912314 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:24.916265 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:49291: BlockingRecv error: recv got EOF from 127.14.227.131:49291 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 03:16:25 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:25 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:25.070972 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:25.057079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:50647)
0924 03:16:25.057694 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:25.057721 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:25.057902 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:25.058306 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:25.058312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:25.058770 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:25.058951 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:25.058962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.059968 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.059975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:25.060978 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:25.060992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.061848 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.061854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:25.062078 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:25.067953 (+ 5875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:25.067970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:25.070651 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":265,"threads_started":1}
W20260924 03:16:25.071107 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:25.057357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:50647 (local address 127.14.227.190:43591)
0924 03:16:25.057640 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:25.057649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:25.057818 (+ 169us) server_negotiation.cc:407] Connection header received
0924 03:16:25.058086 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:25.058090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:25.058155 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:25.058274 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:25.059108 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.059813 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.061131 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.061645 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.061800 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:25.070862 (+ 9062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:50647: BlockingRecv error: recv got EOF from 127.14.227.129:50647 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 03:16:25 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:25 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:25.487485 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:25.478557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:57533)
0924 03:16:25.479081 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:25.479097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:25.479250 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:25.479578 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:25.479583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:25.480138 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:25.480329 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:25.480341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.482026 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.482032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:25.482934 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:25.482946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.483665 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.483670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:25.483789 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:25.484443 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:25.484461 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:25.487209 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":290,"threads_started":1}
W20260924 03:16:25.487767 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:25.478642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:57533 (local address 127.14.227.190:43591)
0924 03:16:25.478833 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:25.478841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:25.479180 (+ 339us) server_negotiation.cc:407] Connection header received
0924 03:16:25.479374 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:25.479387 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:25.479446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:25.479546 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:25.480493 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.481858 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.483060 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.483515 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.483605 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:25.487467 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:57533: BlockingRecv error: recv got EOF from 127.14.227.130:57533 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 03:16:25 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:25 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:25.925633 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:25.917564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:42929)
0924 03:16:25.918093 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:25.918110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:25.918276 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:25.918689 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:25.918694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:25.919121 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:25.919260 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:25.919269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.920202 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.920206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:25.921092 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:25.921102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.921794 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.921802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:25.921991 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:25.922546 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:25.922562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:25.925300 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":261,"threads_started":1}
W20260924 03:16:25.925710 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:25.917716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:42929 (local address 127.14.227.190:43591)
0924 03:16:25.917939 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:25.917967 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:25.918208 (+ 241us) server_negotiation.cc:407] Connection header received
0924 03:16:25.918431 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:25.918435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:25.918513 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:25.918592 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:25.919381 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.920065 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.921202 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:25.921620 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:25.921708 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:25.925487 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:42929: BlockingRecv error: recv got EOF from 127.14.227.131:42929 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 03:16:26 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:26 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:26.079972 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:26.071955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:39741)
0924 03:16:26.072501 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:26.072517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:26.072632 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:26.072930 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:26.072933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:26.073314 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:26.073472 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:26.073480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.074490 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.074494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:26.075429 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:26.075440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.076082 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.076086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:26.076202 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:26.076732 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:26.076748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:26.079626 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":315,"threads_started":1}
W20260924 03:16:26.080057 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:26.072020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:39741 (local address 127.14.227.190:43591)
0924 03:16:26.072178 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:26.072182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:26.072594 (+ 412us) server_negotiation.cc:407] Connection header received
0924 03:16:26.072718 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:26.072722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:26.072777 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:26.072903 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:26.073595 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.074376 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.075545 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.075955 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.076042 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:26.079807 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:39741: BlockingRecv error: recv got EOF from 127.14.227.129:39741 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 03:16:26 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:26 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:26.496565 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:26.488351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:50985)
0924 03:16:26.488816 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:26.488849 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:26.488964 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:26.489235 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:26.489239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:26.489620 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:26.489767 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:26.489774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.490710 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.490714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:26.491763 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:26.491774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.492638 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.492644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:26.492807 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:26.493457 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:26.493475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:26.496219 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
W20260924 03:16:26.496673 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:26.488438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:50985 (local address 127.14.227.190:43591)
0924 03:16:26.488698 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:26.488705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:26.488937 (+ 232us) server_negotiation.cc:407] Connection header received
0924 03:16:26.489058 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:26.489062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:26.489122 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:26.489201 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:26.489902 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.490611 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.491900 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.492491 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.492614 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:26.496436 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:50985: BlockingRecv error: recv got EOF from 127.14.227.130:50985 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 24 03:16:26 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:26 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:26.933939 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:26.926549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:36217)
0924 03:16:26.927069 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:26.927085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:26.927191 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:26.927469 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:26.927473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:26.927821 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:26.927971 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:26.927979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.928869 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.928872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:26.929753 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:26.929762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.930370 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.930374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:26.930559 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:26.931065 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:26.931081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:26.933726 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
W20260924 03:16:26.934230 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:26.926593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:36217 (local address 127.14.227.190:43591)
0924 03:16:26.926741 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:26.926745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:26.927151 (+ 406us) server_negotiation.cc:407] Connection header received
0924 03:16:26.927275 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:26.927278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:26.927335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:26.927411 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:26.928086 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.928765 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.929862 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:26.930255 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:26.930350 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:26.933967 (+ 3617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:36217: BlockingRecv error: recv got EOF from 127.14.227.131:36217 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 03:16:27 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:27 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:27.089622 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:27.081011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:46075)
0924 03:16:27.081545 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:27.081561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:27.081711 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:27.082067 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:27.082072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:27.082455 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:27.082591 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:27.082599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.083600 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.083606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:27.084525 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:27.084536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.085271 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.085276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:27.085447 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:27.086047 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:27.086064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:27.089293 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
W20260924 03:16:27.089730 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:27.081235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:46075 (local address 127.14.227.190:43591)
0924 03:16:27.081479 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:27.081485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:27.081647 (+ 162us) server_negotiation.cc:407] Connection header received
0924 03:16:27.081844 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:27.081851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:27.081913 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:27.082045 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:27.082728 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.083461 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.084671 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.085122 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.085258 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:27.089493 (+ 4235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:46075: BlockingRecv error: recv got EOF from 127.14.227.129:46075 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 03:16:27 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:27 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:27.505648 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:27.497595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:38175)
0924 03:16:27.498198 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:27.498218 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:27.498356 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:27.498786 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:27.498791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:27.499281 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:27.499454 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:27.499462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.500362 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.500367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:27.501250 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:27.501260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.501914 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.501918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:27.502082 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:27.502667 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:27.502685 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:27.505318 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":362,"threads_started":1}
W20260924 03:16:27.505744 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:27.497680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:38175 (local address 127.14.227.190:43591)
0924 03:16:27.497897 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:27.497902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:27.498304 (+ 402us) server_negotiation.cc:407] Connection header received
0924 03:16:27.498536 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:27.498541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:27.498619 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:27.498736 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:27.499577 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.500253 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.501360 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.501783 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.501871 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:27.505493 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:38175: BlockingRecv error: recv got EOF from 127.14.227.130:38175 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 03:16:27 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:27 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:27.943145 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:27.934805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:39741)
0924 03:16:27.935352 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:27.935368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:27.935519 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:27.935836 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:27.935840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:27.936260 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:27.936389 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:27.936396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.937423 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.937430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:27.938420 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:27.938430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.939104 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.939110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:27.939278 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:27.939794 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:27.939809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:27.942824 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
W20260924 03:16:27.943295 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:27.934955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:39741 (local address 127.14.227.190:43591)
0924 03:16:27.935209 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:27.935217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:27.935453 (+ 236us) server_negotiation.cc:407] Connection header received
0924 03:16:27.935638 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:27.935644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:27.935708 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:27.935807 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:27.936545 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.937293 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.938556 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:27.938954 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:27.939092 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:27.943034 (+ 3942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:39741: BlockingRecv error: recv got EOF from 127.14.227.131:39741 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 03:16:28 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:28 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:28.098440 24503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:28.090656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:53051)
0924 03:16:28.091160 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:28.091176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:28.091281 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:28.091577 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:28.091580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:28.091942 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:28.092143 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:28.092151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.093107 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.093110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:28.094166 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:28.094178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.095032 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.095036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:28.095233 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:28.095807 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:28.095825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:28.098232 (+ 2407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
W20260924 03:16:28.098592 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:28.090780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:53051 (local address 127.14.227.190:43591)
0924 03:16:28.090956 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:28.090961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:28.091246 (+ 285us) server_negotiation.cc:407] Connection header received
0924 03:16:28.091383 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:28.091386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:28.091454 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:28.091532 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:28.092259 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.092997 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.094292 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.094856 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.094977 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:28.098390 (+ 3413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:53051: BlockingRecv error: recv got EOF from 127.14.227.129:53051 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 03:16:28 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:28 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:28.514669 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:28.506655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:48783)
0924 03:16:28.507113 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:28.507142 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:28.507266 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:28.507530 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:28.507533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:28.507923 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:28.508096 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:28.508103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.509053 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.509056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:28.509917 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:28.509926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.510676 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.510680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:28.510795 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:28.511394 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:28.511410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:28.514426 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":267,"threads_started":1}
W20260924 03:16:28.514882 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:28.506756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:48783 (local address 127.14.227.190:43591)
0924 03:16:28.507050 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:28.507056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:28.507234 (+ 178us) server_negotiation.cc:407] Connection header received
0924 03:16:28.507346 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:28.507349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:28.507412 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:28.507488 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:28.508226 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.508944 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.510106 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.510554 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.510645 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:28.514631 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:48783: BlockingRecv error: recv got EOF from 127.14.227.130:48783 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 24 03:16:28 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:28 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:28.952283 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:28.944387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:49093)
0924 03:16:28.944877 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:28.944894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:28.944993 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:28.945294 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:28.945297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:28.945672 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:28.945845 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:28.945853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.946823 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.946826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:28.947672 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:28.947682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.948300 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.948304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:28.948440 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:28.949076 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:28.949117 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:28.952018 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":263,"threads_started":1}
W20260924 03:16:28.952316 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:28.944439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:49093 (local address 127.14.227.190:43591)
0924 03:16:28.944603 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:28.944608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:28.944958 (+ 350us) server_negotiation.cc:407] Connection header received
0924 03:16:28.945079 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:28.945082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:28.945143 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:28.945250 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:28.945987 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.946705 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.947799 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:28.948178 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:28.948265 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:28.952150 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:49093: BlockingRecv error: recv got EOF from 127.14.227.131:49093 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 03:16:29 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:29 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:29.107401 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:29.099361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:40217)
0924 03:16:29.099853 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:29.099869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:29.100028 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:29.100367 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:29.100372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:29.100836 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:29.100982 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:29.100990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.102036 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.102043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:29.102939 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:29.102950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.103683 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.103689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:29.103872 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:29.104432 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:29.104449 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:29.107188 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
W20260924 03:16:29.107620 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:29.099414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:40217 (local address 127.14.227.190:43591)
0924 03:16:29.099648 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:29.099657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:29.099958 (+ 301us) server_negotiation.cc:407] Connection header received
0924 03:16:29.100171 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:29.100175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:29.100231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:29.100338 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:29.101123 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.101871 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.103085 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.103536 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.103661 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:29.107362 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:40217: BlockingRecv error: recv got EOF from 127.14.227.129:40217 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 03:16:29 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:29 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:29.523496 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:29.515566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:47725)
0924 03:16:29.516022 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:29.516038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:29.516147 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:29.516431 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:29.516434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:29.516813 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:29.516956 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:29.516963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.518005 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.518010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:29.518910 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:29.518921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.519631 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.519635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:29.519788 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:29.520352 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:29.520369 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:29.523269 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":268,"threads_started":1}
W20260924 03:16:29.523696 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:29.515665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:47725 (local address 127.14.227.190:43591)
0924 03:16:29.515922 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:29.515929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:29.516117 (+ 188us) server_negotiation.cc:407] Connection header received
0924 03:16:29.516242 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:29.516246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:29.516306 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:29.516385 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:29.517066 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.517775 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.519065 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.519497 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.519586 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:29.523459 (+ 3873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:47725: BlockingRecv error: recv got EOF from 127.14.227.130:47725 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 24 03:16:29 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:29 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:29.961198 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:29.953255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:44409)
0924 03:16:29.953767 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:29.953784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:29.953890 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:29.954217 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:29.954221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:29.954578 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:29.954739 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:29.954754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.955741 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.955745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:29.956633 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:29.956643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.957265 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.957269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:29.957388 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:29.957982 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:29.958007 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:29.960856 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":300,"threads_started":1}
W20260924 03:16:29.961287 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:29.953323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:44409 (local address 127.14.227.190:43591)
0924 03:16:29.953482 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:29.953487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:29.953858 (+ 371us) server_negotiation.cc:407] Connection header received
0924 03:16:29.953998 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:29.954002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:29.954058 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:29.954152 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:29.954900 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.955620 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.956747 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:29.957149 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:29.957237 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:29.961041 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:44409: BlockingRecv error: recv got EOF from 127.14.227.131:44409 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 03:16:30 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:30 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:30.120278 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:30.108282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:56225)
0924 03:16:30.108837 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:30.108851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:30.108996 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:30.109362 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:30.109367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:30.109768 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:30.109915 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:30.109923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.111200 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.111208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:30.112431 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:30.112443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.113177 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.113182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:30.113353 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:30.117172 (+ 3819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:30.117206 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:30.119934 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":284,"threads_started":1}
W20260924 03:16:30.120332 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:30.108427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:56225 (local address 127.14.227.190:43591)
0924 03:16:30.108657 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:30.108665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:30.108933 (+ 268us) server_negotiation.cc:407] Connection header received
0924 03:16:30.109134 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:30.109139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:30.109209 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:30.109330 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:30.110078 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.111075 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.112573 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.112997 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.113129 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:30.120104 (+ 6975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:56225: BlockingRecv error: recv got EOF from 127.14.227.129:56225 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 03:16:30 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:30 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:30.532595 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:30.524380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:47653)
0924 03:16:30.524930 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:30.524946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:30.525079 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:30.525369 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:30.525372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:30.525760 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:30.525907 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:30.525914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.526926 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.526930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:30.527819 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:30.527829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.528464 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.528468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:30.528572 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:30.529110 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:30.529126 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:30.532263 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
W20260924 03:16:30.532696 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:30.524546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:47653 (local address 127.14.227.190:43591)
0924 03:16:30.524714 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:30.524718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:30.525014 (+ 296us) server_negotiation.cc:407] Connection header received
0924 03:16:30.525188 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:30.525191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:30.525248 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:30.525328 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:30.526071 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.526811 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.527936 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.528321 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.528408 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:30.532458 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:47653: BlockingRecv error: recv got EOF from 127.14.227.130:47653 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 03:16:30 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:30 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:30.970250 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:30.962125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:57543)
0924 03:16:30.962633 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:30.962648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:30.962807 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:30.963132 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:30.963137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:30.963549 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:30.963692 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:30.963700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.964690 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.964696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:30.965634 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:30.965645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.966490 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.966497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:30.966671 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:30.967361 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:30.967377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:30.970024 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":249,"threads_started":1}
W20260924 03:16:30.970464 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:30.962130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:57543 (local address 127.14.227.190:43591)
0924 03:16:30.962269 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:30.962274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:30.962741 (+ 467us) server_negotiation.cc:407] Connection header received
0924 03:16:30.962948 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:30.962952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:30.963008 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:30.963104 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:30.963837 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.964558 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.965785 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:30.966262 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:30.966449 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:30.970217 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:57543: BlockingRecv error: recv got EOF from 127.14.227.131:57543 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 03:16:31 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:31 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:31.129585 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:31.121258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:40097)
0924 03:16:31.121722 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:31.121737 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:31.121903 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:31.122262 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:31.122267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:31.122673 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:31.122840 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:31.122849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.123883 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.123888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:31.124766 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:31.124779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.125608 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.125613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:31.125788 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:31.126511 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:31.126528 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:31.129223 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
W20260924 03:16:31.129683 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:31.121316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:40097 (local address 127.14.227.190:43591)
0924 03:16:31.121579 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:31.121586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:31.121823 (+ 237us) server_negotiation.cc:407] Connection header received
0924 03:16:31.122046 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:31.122058 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:31.122129 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:31.122230 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:31.122970 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.123739 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.124907 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.125443 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.125600 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:31.129427 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:40097: BlockingRecv error: recv got EOF from 127.14.227.129:40097 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 24 03:16:31 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:31 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:31.543560 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:31.533616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:47189)
0924 03:16:31.534138 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:31.534168 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:31.534301 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:31.534619 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:31.534626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:31.535143 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:31.535401 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:31.535415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.536483 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.536493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:31.538282 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:31.538298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.539104 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.539113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:31.539336 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:31.539995 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:31.540025 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:31.543217 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":265,"threads_started":1}
W20260924 03:16:31.543599 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:31.533725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:47189 (local address 127.14.227.190:43591)
0924 03:16:31.533933 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:31.533939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:31.534235 (+ 296us) server_negotiation.cc:407] Connection header received
0924 03:16:31.534409 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:31.534412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:31.534485 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:31.534597 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:31.535549 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.536337 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.538430 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.538959 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.539048 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:31.543394 (+ 4346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:47189: BlockingRecv error: recv got EOF from 127.14.227.130:47189 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 03:16:31 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:31 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.131@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:31.978804 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:31.971177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.131:51907)
0924 03:16:31.971682 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:31.971706 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:31.971815 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:31.972089 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:31.972092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:31.972440 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:31.972609 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:31.972617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.973504 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.973507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:31.974392 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:31.974401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.975052 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.975056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:31.975175 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:31.975758 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:31.975775 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:31.978557 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":253,"threads_started":1}
W20260924 03:16:31.978991 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:31.971247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:51907 (local address 127.14.227.190:43591)
0924 03:16:31.971437 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:31.971446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:31.971785 (+ 339us) server_negotiation.cc:407] Connection header received
0924 03:16:31.971914 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:31.971917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:31.971975 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:31.972048 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:31.972724 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.973391 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.974500 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:31.974908 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:31.975022 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:31.978777 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.131:51907: BlockingRecv error: recv got EOF from 127.14.227.131:51907 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 03:16:32 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:32 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.129@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:32.139401 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:32.130979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.129:46367)
0924 03:16:32.131510 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:32.131526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:32.131667 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:32.132022 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:32.132027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:32.132408 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:32.132557 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:32.132564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.133651 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.133658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:32.134946 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:32.134958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.135737 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.135743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:32.135914 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:32.136456 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:32.136472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:32.139046 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
W20260924 03:16:32.139469 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:32.131006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:46367 (local address 127.14.227.190:43591)
0924 03:16:32.131168 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:32.131172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:32.131605 (+ 433us) server_negotiation.cc:407] Connection header received
0924 03:16:32.131787 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:32.131793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:32.131882 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:32.131990 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:32.132691 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.133505 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.135078 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.135595 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.135731 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:32.139239 (+ 3508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.129:46367: BlockingRecv error: recv got EOF from 127.14.227.129:46367 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 03:16:32 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 03:16:32 dist-test-slave-vd0p krb5kdc[23471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.227.130@KRBTEST.COM for oryx/127.14.227.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 03:16:32.553041 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:32.544616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43591 (local address 127.14.227.130:36535)
0924 03:16:32.545081 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:32.545096 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:32.545201 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:32.545471 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:32.545474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:32.545851 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:32.546168 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:32.546178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.547122 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.547125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:32.548140 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:32.548155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.549096 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.549118 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:32.549284 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:32.549909 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:32.549927 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:32.552685 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.227.190:43591: Server oryx/127.14.227.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":265,"threads_started":1}
W20260924 03:16:32.553140 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:32.544655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:36535 (local address 127.14.227.190:43591)
0924 03:16:32.544868 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:32.544878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:32.545170 (+ 292us) server_negotiation.cc:407] Connection header received
0924 03:16:32.545292 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:32.545296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:32.545353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:32.545438 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:32.546282 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.547005 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.548283 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.548945 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.549085 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:32.552885 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.227.130:36535: BlockingRecv error: recv got EOF from 127.14.227.130:36535 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:16:32.832181 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24058
I20260924 03:16:32.843880 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:32.833108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.14.227.129:33975)
0924 03:16:32.833639 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:32.833645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:32.833670 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:16:32.833724 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:32.833730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:32.834050 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:32.834198 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:32.835529 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.836757 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.837870 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.838465 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.838570 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:32.840769 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:32.840777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:32.840789 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:32.840816 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:32.843384 (+ 2568us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.227.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:16:32.843624 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54672: Request ticket server kudu/127.14.227.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":286,"threads_started":1}
W20260924 03:16:32.843982 24518 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54672: Request ticket server kudu/127.14.227.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:16:32.843999 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24058: Not authorized: Client connection negotiation failed: client connection to 127.14.227.129:33975: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.227.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:16:32.844213 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 24058
I20260924 03:16:32.860795 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24209
I20260924 03:16:32.872134 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:32.861801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.14.227.130:35697)
0924 03:16:32.862435 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:32.862443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:32.862472 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:16:32.862528 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:32.862534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:32.862746 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:32.862889 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:32.864047 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.865090 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.866338 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.866885 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.866984 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:32.869332 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:32.869340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:32.869353 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:32.869383 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:32.871643 (+ 2260us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.227.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:16:32.871868 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47960: Request ticket server kudu/127.14.227.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":326,"threads_started":1}
W20260924 03:16:32.872237 24519 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47960: Request ticket server kudu/127.14.227.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:16:32.872277 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24209: Not authorized: Client connection negotiation failed: client connection to 127.14.227.130:35697: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.227.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:16:32.872449 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 24209
I20260924 03:16:32.888405 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24340
I20260924 03:16:32.899528 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:32.889314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.14.227.131:43379)
0924 03:16:32.889861 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:32.889868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:32.889893 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:16:32.890003 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:32.890017 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:32.890238 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:32.890390 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:32.891547 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.892553 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.893660 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:32.894294 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:32.894387 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:32.896745 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:32.896752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:32.896766 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:32.896791 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:32.899033 (+ 2242us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.227.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 03:16:32.899258 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52688: Request ticket server kudu/127.14.227.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":267,"threads_started":1}
W20260924 03:16:32.899645 24520 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52688: Request ticket server kudu/127.14.227.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 03:16:32.899641 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24340: Not authorized: Client connection negotiation failed: client connection to 127.14.227.131:43379: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.227.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 03:16:32.899828 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 24340
I20260924 03:16:32.916136 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 23989
I20260924 03:16:32.917435 24051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:33.031545 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 23989
2026-09-24T03:16:33Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20817 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T03:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:16:33Z Disabled control of system clock
I20260924 03:16:33.103952 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:34657
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:34657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 03:16:33.426163 24525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:33.426440 24525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:33.426592 24525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:33.433883 24525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:16:33.434021 24525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:33.434114 24525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:33.434212 24525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:16:33.434291 24525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:16:33.443537 24525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:34657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:34657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:33.445171 24525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:33.447306 24525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:33.459327 24530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:33.459295 24531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:33.459355 24533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:33.459882 24525 server_base.cc:1034] running on GCE node
I20260924 03:16:33.460629 24525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:33.461905 24525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:33.463181 24525 hybrid_clock.cc:648] HybridClock initialized: now 1790219793463137 us; error 34 us; skew 500 ppm
I20260924 03:16:33.463503 24525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:16:33.465601 24525 webserver.cc:460] Webserver started at http://127.14.227.190:34091/ using document root <none> and password file <none>
I20260924 03:16:33.466310 24525 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:33.466396 24525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:33.466657 24525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:33.469316 24525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "186fd57694934a85afc6015884c732c5"
format_stamp: "Formatted at 2026-09-24 03:16:33 on dist-test-slave-vd0p"
server_key: "09b5fd6874d8ece6e12535f6b7dae811"
server_key_iv: "f74b8fcddff628caeb7d5ea1f5ed864a"
server_key_version: "encryptionkey@0"
I20260924 03:16:33.470062 24525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "186fd57694934a85afc6015884c732c5"
format_stamp: "Formatted at 2026-09-24 03:16:33 on dist-test-slave-vd0p"
server_key: "09b5fd6874d8ece6e12535f6b7dae811"
server_key_iv: "f74b8fcddff628caeb7d5ea1f5ed864a"
server_key_version: "encryptionkey@0"
I20260924 03:16:33.474819 24525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:16:33.478034 24539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:33.479296 24525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:16:33.479429 24525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "186fd57694934a85afc6015884c732c5"
format_stamp: "Formatted at 2026-09-24 03:16:33 on dist-test-slave-vd0p"
server_key: "09b5fd6874d8ece6e12535f6b7dae811"
server_key_iv: "f74b8fcddff628caeb7d5ea1f5ed864a"
server_key_version: "encryptionkey@0"
I20260924 03:16:33.479604 24525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:33.524492 24525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:33.525300 24525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:33.525542 24525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:33.542438 24525 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:34657
I20260924 03:16:33.542461 24590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:34657 every 8 connection(s)
I20260924 03:16:33.543738 24525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:16:33.547451 24591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:33.553239 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 24525
I20260924 03:16:33.553427 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:16:33.554255 15246 external_mini_cluster.cc:1428] Setting key 239fd7425ef2c6cccb0f1fdc9df0c23b
I20260924 03:16:33.558236 24591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5: Bootstrap starting.
I20260924 03:16:33.561514 24591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:33.562772 24591 log.cc:824] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:33.565824 24591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5: No bootstrap required, opened a new log
I20260924 03:16:33.568027 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:33.556316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.14.227.190:34657)
0924 03:16:33.557280 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:33.557291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:33.557331 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:16:33.559017 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:33.559039 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:33.559483 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:33.559828 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:33.560501 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:33.561782 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:33.563642 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:33.564453 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:33.564581 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:33.565744 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:33.565751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:33.565763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:33.566357 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:33.566433 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:33.566453 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:33.566533 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:33.566764 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 03:16:33.567081 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":350,"threads_started":1}
I20260924 03:16:33.573256 24591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fd57694934a85afc6015884c732c5" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34657 } }
I20260924 03:16:33.573601 24591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:33.573714 24591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186fd57694934a85afc6015884c732c5, State: Initialized, Role: FOLLOWER
I20260924 03:16:33.574301 24591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fd57694934a85afc6015884c732c5" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34657 } }
I20260924 03:16:33.574566 24591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:16:33.574723 24591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:16:33.574892 24591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:33.577018 24591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fd57694934a85afc6015884c732c5" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34657 } }
I20260924 03:16:33.577499 24591 leader_election.cc:302] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 186fd57694934a85afc6015884c732c5; no voters:
I20260924 03:16:33.578082 24591 leader_election.cc:288] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:16:33.578154 24596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:33.578889 24596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [term 1 LEADER]: Becoming Leader. State: Replica: 186fd57694934a85afc6015884c732c5, State: Running, Role: LEADER
I20260924 03:16:33.579412 24596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fd57694934a85afc6015884c732c5" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34657 } }
I20260924 03:16:33.580039 24591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:16:33.582590 24598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 186fd57694934a85afc6015884c732c5. Latest consensus state: current_term: 1 leader_uuid: "186fd57694934a85afc6015884c732c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fd57694934a85afc6015884c732c5" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34657 } } }
I20260924 03:16:33.582948 24598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:33.582751 24597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "186fd57694934a85afc6015884c732c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186fd57694934a85afc6015884c732c5" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34657 } } }
I20260924 03:16:33.583732 24597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:33.583854 24604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:16:33.589385 24604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:16:33.598367 24604 catalog_manager.cc:1347] Generated new cluster ID: 4244fc16cd8e4a93b99c40cd1cb7432e
I20260924 03:16:33.598486 24604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:16:33.623077 24604 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:16:33.624383 24604 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:16:33.638556 24604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 186fd57694934a85afc6015884c732c5: Generated new TSK 0
I20260924 03:16:33.639407 24604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:16:33.652899 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34657
--builtin_ntp_servers=127.14.227.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 03:16:33.973122 24615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:33.973430 24615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:33.973618 24615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:33.981477 24615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:33.981573 24615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:33.981737 24615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:16:33.991112 24615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:33.992334 24615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:33.994063 24615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:34.006812 24621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:34.006781 24620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:34.007404 24623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:34.007596 24615 server_base.cc:1034] running on GCE node
I20260924 03:16:34.008173 24615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:34.009068 24615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:34.010262 24615 hybrid_clock.cc:648] HybridClock initialized: now 1790219794010223 us; error 29 us; skew 500 ppm
I20260924 03:16:34.010614 24615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:16:34.012707 24615 webserver.cc:460] Webserver started at http://127.14.227.129:42575/ using document root <none> and password file <none>
I20260924 03:16:34.013388 24615 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:34.013469 24615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:34.013813 24615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:34.016566 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "80adfe36af15467dae26eb73e97ac036"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "0f12833ae4d4a870c03b0c806ccea39a"
server_key_iv: "03b6ce3461997fd100bd851a5d0d7d19"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.017334 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "80adfe36af15467dae26eb73e97ac036"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "0f12833ae4d4a870c03b0c806ccea39a"
server_key_iv: "03b6ce3461997fd100bd851a5d0d7d19"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.022023 24615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 03:16:34.025146 24629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.026605 24615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:16:34.026784 24615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "80adfe36af15467dae26eb73e97ac036"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "0f12833ae4d4a870c03b0c806ccea39a"
server_key_iv: "03b6ce3461997fd100bd851a5d0d7d19"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.026989 24615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:34.041620 24615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:34.042443 24615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:34.042729 24615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:34.043637 24615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:34.044951 24615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:34.045077 24615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.045203 24615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:34.045288 24615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.077670 24615 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:43455
I20260924 03:16:34.077754 24741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:43455 every 8 connection(s)
I20260924 03:16:34.078974 24615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:16:34.083855 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 24615
I20260924 03:16:34.084075 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:16:34.084709 15246 external_mini_cluster.cc:1428] Setting key 2538a910cefe825aea1126aa46e489b0
I20260924 03:16:34.087517 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34657
--builtin_ntp_servers=127.14.227.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:16:34.092832 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:34.081751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:57553 (local address 127.14.227.190:34657)
0924 03:16:34.082229 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:34.082235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:34.083519 (+ 1284us) server_negotiation.cc:407] Connection header received
0924 03:16:34.085838 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:34.085842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:34.085901 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:34.086055 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:34.087196 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.088317 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.089786 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.090521 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.090652 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:34.091899 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:34.091907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:34.091914 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:34.091946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:34.092002 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:34.092020 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:34.092106 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:34.092293 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 03:16:34.092394 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":241,"threads_started":1}
I20260924 03:16:34.093490 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:34.082376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34657 (local address 127.14.227.129:57553)
0924 03:16:34.083358 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:34.083393 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:34.085555 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:34.086278 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:34.086290 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:34.086402 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:34.087023 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:34.087048 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.088457 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.088461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:34.089638 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:34.089650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.090700 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.090705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:34.090922 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:34.091602 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:16:34.091605 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:34.091641 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:16:34.091727 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:34.092109 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:34.092116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:34.092121 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:16:34.092249 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 03:16:34.092508 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":292,"threads_started":1}
I20260924 03:16:34.095103 24742 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34657
I20260924 03:16:34.095389 24742 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:34.096097 24742 heartbeater.cc:511] Master 127.14.227.190:34657 requested a full tablet report, sending...
I20260924 03:16:34.097908 24556 ts_manager.cc:194] Registered new tserver with Master: 80adfe36af15467dae26eb73e97ac036 (127.14.227.129:43455)
I20260924 03:16:34.099319 24556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.129:57553
W20260924 03:16:34.399307 24746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:34.399649 24746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:34.399787 24746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:34.406719 24746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:34.406833 24746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:34.407003 24746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:16:34.415433 24746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:34.416642 24746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:34.418365 24746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:34.430831 24751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:34.430809 24752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:34.430852 24754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:34.432122 24746 server_base.cc:1034] running on GCE node
I20260924 03:16:34.432765 24746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:34.433650 24746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:34.434849 24746 hybrid_clock.cc:648] HybridClock initialized: now 1790219794434786 us; error 59 us; skew 500 ppm
I20260924 03:16:34.435118 24746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:16:34.437331 24746 webserver.cc:460] Webserver started at http://127.14.227.130:43171/ using document root <none> and password file <none>
I20260924 03:16:34.438025 24746 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:34.438118 24746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:34.438385 24746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:34.441236 24746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "375d3f91a7b1479fb7fe878a545d0090"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "15d53f2aab215cc96f560a8301a58c99"
server_key_iv: "1285b13aeaacc3b5ac32659049e98500"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.442013 24746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "375d3f91a7b1479fb7fe878a545d0090"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "15d53f2aab215cc96f560a8301a58c99"
server_key_iv: "1285b13aeaacc3b5ac32659049e98500"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.446805 24746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:16:34.450788 24760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.452093 24746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:16:34.452273 24746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "375d3f91a7b1479fb7fe878a545d0090"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "15d53f2aab215cc96f560a8301a58c99"
server_key_iv: "1285b13aeaacc3b5ac32659049e98500"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.452482 24746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:34.475010 24746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:34.475841 24746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:34.476140 24746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:34.477119 24746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:34.478547 24746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:34.478662 24746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.478787 24746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:34.478868 24746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.511253 24746 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:40797
I20260924 03:16:34.511340 24872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:40797 every 8 connection(s)
I20260924 03:16:34.512516 24746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:16:34.518658 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 24746
I20260924 03:16:34.518842 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:16:34.519315 15246 external_mini_cluster.cc:1428] Setting key 3fff1500810b76e3457c20a92b8fa6b3
I20260924 03:16:34.521914 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34657
--builtin_ntp_servers=127.14.227.148:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 03:16:34.524866 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:34.514997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:34177 (local address 127.14.227.190:34657)
0924 03:16:34.515210 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:34.515216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:34.516585 (+ 1369us) server_negotiation.cc:407] Connection header received
0924 03:16:34.518319 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:34.518328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:34.518422 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:34.518545 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:34.519663 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.520599 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.522250 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.522629 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.522763 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:34.524040 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:34.524051 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:34.524054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:34.524069 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:34.524098 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:34.524109 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:34.524201 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:34.524415 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 03:16:34.524534 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:16:34.525743 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:34.515563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34657 (local address 127.14.227.130:34177)
0924 03:16:34.516441 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:34.516474 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:34.518051 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:34.518725 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:34.518733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:34.518868 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:34.519505 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:34.519522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.520779 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.520784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:34.522040 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:34.522072 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.522838 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.522845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:34.523079 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:34.523748 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:16:34.523750 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:34.523787 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:16:34.523861 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:34.524201 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:34.524213 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:34.524223 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:16:34.524395 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 03:16:34.524771 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":270,"threads_started":1}
I20260924 03:16:34.527063 24873 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34657
I20260924 03:16:34.527359 24873 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:34.528018 24873 heartbeater.cc:511] Master 127.14.227.190:34657 requested a full tablet report, sending...
I20260924 03:16:34.529268 24556 ts_manager.cc:194] Registered new tserver with Master: 375d3f91a7b1479fb7fe878a545d0090 (127.14.227.130:40797)
I20260924 03:16:34.530114 24556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.130:34177
W20260924 03:16:34.842552 24876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:34.842805 24876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:34.842983 24876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:34.850061 24876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:34.850152 24876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:34.850306 24876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:16:34.859237 24876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:34.860427 24876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:34.862334 24876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:34.874828 24881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:34.874878 24882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:34.874909 24884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:34.875664 24876 server_base.cc:1034] running on GCE node
I20260924 03:16:34.876238 24876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:34.877103 24876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:34.878289 24876 hybrid_clock.cc:648] HybridClock initialized: now 1790219794878239 us; error 42 us; skew 500 ppm
I20260924 03:16:34.878587 24876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:16:34.880719 24876 webserver.cc:460] Webserver started at http://127.14.227.131:44677/ using document root <none> and password file <none>
I20260924 03:16:34.881376 24876 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:34.881458 24876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:34.881717 24876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:34.884699 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "2e3ebb0803c94320a7852345bc54ea5e"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "0f8c432d3e00d6545f858e71cdb38a7b"
server_key_iv: "7c46da26032b9474fd5f5ebdb7e1751d"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.885507 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "2e3ebb0803c94320a7852345bc54ea5e"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "0f8c432d3e00d6545f858e71cdb38a7b"
server_key_iv: "7c46da26032b9474fd5f5ebdb7e1751d"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.890916 24876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:16:34.894145 24890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.895401 24876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:16:34.895524 24876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "2e3ebb0803c94320a7852345bc54ea5e"
format_stamp: "Formatted at 2026-09-24 03:16:34 on dist-test-slave-vd0p"
server_key: "0f8c432d3e00d6545f858e71cdb38a7b"
server_key_iv: "7c46da26032b9474fd5f5ebdb7e1751d"
server_key_version: "encryptionkey@0"
I20260924 03:16:34.895702 24876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:34.931761 24876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:34.932641 24876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:34.932898 24876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:34.933780 24876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:34.935178 24876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:34.935263 24876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.935336 24876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:34.935405 24876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:34.967756 24876 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:41341
I20260924 03:16:34.967800 25002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:41341 every 8 connection(s)
I20260924 03:16:34.968983 24876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:16:34.973052 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 24876
I20260924 03:16:34.973213 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:16:34.973770 15246 external_mini_cluster.cc:1428] Setting key 25a66907142afc7e75afa45be799a051
I20260924 03:16:34.981698 24744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:34.971642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:53191 (local address 127.14.227.190:34657)
0924 03:16:34.971846 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:34.971852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:34.973411 (+ 1559us) server_negotiation.cc:407] Connection header received
0924 03:16:34.975536 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:34.975542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:34.975640 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:34.975726 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:34.976904 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.977648 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.979242 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.979631 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.979721 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:34.981057 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:34.981061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:34.981064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:34.981080 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:34.981111 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:34.981122 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:34.981187 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:34.981378 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:16:34.981479 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:16:34.982762 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:34.972216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34657 (local address 127.14.227.131:53191)
0924 03:16:34.973244 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:34.973295 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:34.975236 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:34.975948 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:34.975956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:34.976039 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:34.976727 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:34.976755 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.977794 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.977798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:34.979125 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:34.979136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.979801 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:34.979805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:34.980072 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:34.980749 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:16:34.980752 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:34.980787 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:16:34.980863 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:34.981204 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:34.981212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:34.981218 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:16:34.981354 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 03:16:34.981637 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":247,"threads_started":1}
I20260924 03:16:34.983978 25003 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34657
I20260924 03:16:34.984287 25003 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:34.985001 25003 heartbeater.cc:511] Master 127.14.227.190:34657 requested a full tablet report, sending...
I20260924 03:16:34.986359 24556 ts_manager.cc:194] Registered new tserver with Master: 2e3ebb0803c94320a7852345bc54ea5e (127.14.227.131:41341)
I20260924 03:16:34.987146 24556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.131:53191
I20260924 03:16:34.988937 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:16:34.994822 24744 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:34.993332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.14.227.190:34657)
0924 03:16:34.993471 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:34.993475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:34.993881 (+ 406us) server_negotiation.cc:407] Connection header received
0924 03:16:34.994057 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:34.994061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:34.994128 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:34.994221 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:34.994576 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53806: BlockingRecv error: recv got EOF from 127.0.0.1:53806 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:16:34.996117 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24615
I20260924 03:16:35.003230 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:34.996882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.14.227.129:43455)
0924 03:16:34.997380 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:34.997387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:34.997410 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:16:34.997466 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:34.997471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:34.997651 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:34.997765 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:34.998166 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:34.999097 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:35.000316 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:35.000833 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:35.000928 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:35.001857 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:35.001864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:35.001874 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:35.002485 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:35.002531 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:35.002549 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:35.002631 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:35.002768 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 03:16:35.002903 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":271,"threads_started":1}
I20260924 03:16:35.004307 24737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:35.090129 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 24615
I20260924 03:16:35.107311 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24746
I20260924 03:16:35.114914 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:35.108278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.14.227.130:40797)
0924 03:16:35.108845 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:35.108852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:35.108876 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:16:35.108931 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:35.108937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:35.109145 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:35.109265 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:35.109604 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:35.110608 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:35.111909 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:35.112432 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:35.112522 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:35.113552 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:35.113559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:35.113572 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:35.114222 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:35.114266 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:35.114285 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:35.114370 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:35.114515 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 03:16:35.114627 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":309,"threads_started":1}
I20260924 03:16:35.116053 24868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:35.197748 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 24746
I20260924 03:16:35.213790 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24876
I20260924 03:16:35.227385 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:35.214755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.14.227.131:41341)
0924 03:16:35.215390 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:35.215398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:35.215428 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:16:35.215486 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:35.215492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:35.215721 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:35.215898 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:35.216344 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:35.217371 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:35.218656 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:35.219218 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:35.219354 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:35.225904 (+ 6550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:35.225913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:35.225927 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:35.226566 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:35.226612 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:35.226631 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:35.226754 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:35.226911 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:16:35.227036 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":334,"threads_started":1}
I20260924 03:16:35.228606 24998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:35.308444 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 24876
I20260924 03:16:35.323637 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 24525
I20260924 03:16:35.324774 24586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:35.428757 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 24525
2026-09-24T03:16:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2392 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T03:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:16:35Z Disabled control of system clock
I20260924 03:16:35.496812 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:34537
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:34537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:16:35.810274 25022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:35.810535 25022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:35.810689 25022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:35.817736 25022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:16:35.817831 25022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:35.817942 25022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:35.818061 25022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:16:35.818116 25022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:16:35.826924 25022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:34537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:35.828166 25022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:35.829988 25022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:35.841468 25027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:35.841423 25028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:35.841650 25030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:35.842694 25022 server_base.cc:1034] running on GCE node
I20260924 03:16:35.843405 25022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:35.844772 25022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:35.846098 25022 hybrid_clock.cc:648] HybridClock initialized: now 1790219795846058 us; error 49 us; skew 500 ppm
I20260924 03:16:35.846438 25022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:16:35.848567 25022 webserver.cc:460] Webserver started at http://127.14.227.190:36709/ using document root <none> and password file <none>
I20260924 03:16:35.849233 25022 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:35.849356 25022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:35.849697 25022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:35.852550 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "a570b83d814c4f8c9302b181db59bd63"
format_stamp: "Formatted at 2026-09-24 03:16:35 on dist-test-slave-vd0p"
server_key: "9d75f007d3685841726c0f6c17d27cd0"
server_key_iv: "2964e36a94be3dd21dd96ca8d7a2b05b"
server_key_version: "encryptionkey@0"
I20260924 03:16:35.853332 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "a570b83d814c4f8c9302b181db59bd63"
format_stamp: "Formatted at 2026-09-24 03:16:35 on dist-test-slave-vd0p"
server_key: "9d75f007d3685841726c0f6c17d27cd0"
server_key_iv: "2964e36a94be3dd21dd96ca8d7a2b05b"
server_key_version: "encryptionkey@0"
I20260924 03:16:35.858325 25022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 03:16:35.861513 25036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:35.862788 25022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:16:35.862951 25022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "a570b83d814c4f8c9302b181db59bd63"
format_stamp: "Formatted at 2026-09-24 03:16:35 on dist-test-slave-vd0p"
server_key: "9d75f007d3685841726c0f6c17d27cd0"
server_key_iv: "2964e36a94be3dd21dd96ca8d7a2b05b"
server_key_version: "encryptionkey@0"
I20260924 03:16:35.863157 25022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:35.868687 25022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:35.869499 25022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:35.869800 25022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:35.887326 25022 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:34537
I20260924 03:16:35.887359 25087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:34537 every 8 connection(s)
I20260924 03:16:35.888644 25022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:16:35.892182 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:35.895660 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 25022
I20260924 03:16:35.895871 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:16:35.896476 15246 external_mini_cluster.cc:1428] Setting key b75fda2df942726b584625463df856fa
I20260924 03:16:35.902813 25088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63: Bootstrap starting.
I20260924 03:16:35.904831 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:35.898404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.14.227.190:34537)
0924 03:16:35.899290 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:35.899300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:35.899340 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:16:35.901409 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:35.901430 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:35.901813 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:35.902245 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:35.903320 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:35.903326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:35.903339 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:35.903352 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:35.903405 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:35.903420 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:35.903632 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:35.903764 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 03:16:35.903993 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":328,"threads_started":1}
I20260924 03:16:35.905848 25088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:35.907155 25088 log.cc:824] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:35.910019 25088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63: No bootstrap required, opened a new log
I20260924 03:16:35.916716 25088 raft_consensus.cc:348] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a570b83d814c4f8c9302b181db59bd63" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34537 } }
I20260924 03:16:35.917160 25088 raft_consensus.cc:374] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:35.917236 25088 raft_consensus.cc:729] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a570b83d814c4f8c9302b181db59bd63, State: Initialized, Role: FOLLOWER
I20260924 03:16:35.917698 25088 consensus_queue.cc:260] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a570b83d814c4f8c9302b181db59bd63" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34537 } }
I20260924 03:16:35.917877 25088 raft_consensus.cc:388] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:16:35.918020 25088 raft_consensus.cc:482] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:16:35.918143 25088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:35.920084 25088 raft_consensus.cc:504] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a570b83d814c4f8c9302b181db59bd63" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34537 } }
I20260924 03:16:35.920506 25088 leader_election.cc:302] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a570b83d814c4f8c9302b181db59bd63; no voters:
I20260924 03:16:35.921028 25088 leader_election.cc:288] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:16:35.921113 25093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:35.921898 25093 raft_consensus.cc:686] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [term 1 LEADER]: Becoming Leader. State: Replica: a570b83d814c4f8c9302b181db59bd63, State: Running, Role: LEADER
I20260924 03:16:35.922557 25093 consensus_queue.cc:237] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a570b83d814c4f8c9302b181db59bd63" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34537 } }
I20260924 03:16:35.923303 25088 sys_catalog.cc:564] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:16:35.925325 25095 sys_catalog.cc:455] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a570b83d814c4f8c9302b181db59bd63. Latest consensus state: current_term: 1 leader_uuid: "a570b83d814c4f8c9302b181db59bd63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a570b83d814c4f8c9302b181db59bd63" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34537 } } }
I20260924 03:16:35.925302 25094 sys_catalog.cc:455] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a570b83d814c4f8c9302b181db59bd63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a570b83d814c4f8c9302b181db59bd63" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34537 } } }
I20260924 03:16:35.925756 25094 sys_catalog.cc:458] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:35.926657 25095 sys_catalog.cc:458] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63 [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:35.928900 25099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:16:35.933984 25099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:16:35.941901 25099 catalog_manager.cc:1347] Generated new cluster ID: 7fcc4ba9e7724e3cb33492c1ecbbaa4e
I20260924 03:16:35.942016 25099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:16:35.955364 25099 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:16:35.955756 25099 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:16:35.961596 25099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a570b83d814c4f8c9302b181db59bd63: Generated new TSK 0
I20260924 03:16:35.962358 25099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:16:35.970109 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34537
--builtin_ntp_servers=127.14.227.148:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:16:36.319716 25112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:36.320053 25112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:36.320232 25112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:36.327962 25112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:36.328116 25112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:36.328310 25112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:16:36.337644 25112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:36.339008 25112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:36.340792 25112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:36.353451 25117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:36.353510 25118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:36.353542 25120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:36.354331 25112 server_base.cc:1034] running on GCE node
I20260924 03:16:36.354913 25112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:36.355847 25112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:36.357072 25112 hybrid_clock.cc:648] HybridClock initialized: now 1790219796357029 us; error 55 us; skew 500 ppm
I20260924 03:16:36.357424 25112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:16:36.359730 25112 webserver.cc:460] Webserver started at http://127.14.227.129:44347/ using document root <none> and password file <none>
I20260924 03:16:36.360397 25112 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:36.360524 25112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:36.360862 25112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:36.363749 25112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "0ca69f7ba3964e6e98c9ddee91dc061b"
format_stamp: "Formatted at 2026-09-24 03:16:36 on dist-test-slave-vd0p"
server_key: "3340053469a7efa650105c722cbe92da"
server_key_iv: "e57c6df942eddd67fcba0d548a870db7"
server_key_version: "encryptionkey@0"
I20260924 03:16:36.364527 25112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "0ca69f7ba3964e6e98c9ddee91dc061b"
format_stamp: "Formatted at 2026-09-24 03:16:36 on dist-test-slave-vd0p"
server_key: "3340053469a7efa650105c722cbe92da"
server_key_iv: "e57c6df942eddd67fcba0d548a870db7"
server_key_version: "encryptionkey@0"
I20260924 03:16:36.369613 25112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:16:36.372936 25126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:36.374313 25112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:16:36.374433 25112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "0ca69f7ba3964e6e98c9ddee91dc061b"
format_stamp: "Formatted at 2026-09-24 03:16:36 on dist-test-slave-vd0p"
server_key: "3340053469a7efa650105c722cbe92da"
server_key_iv: "e57c6df942eddd67fcba0d548a870db7"
server_key_version: "encryptionkey@0"
I20260924 03:16:36.374605 25112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:36.379858 25112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:36.380568 25112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:36.380805 25112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:36.381595 25112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:36.382987 25112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:36.383070 25112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:36.383149 25112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:36.383203 25112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:36.423681 25112 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:39681
I20260924 03:16:36.423698 25238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:39681 every 8 connection(s)
I20260924 03:16:36.425333 25112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:16:36.427671 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 25112
I20260924 03:16:36.427899 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:16:36.428743 15246 external_mini_cluster.cc:1428] Setting key 196a2f1e438dc58c7a3a76580694b8f0
I20260924 03:16:36.431648 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34537
--builtin_ntp_servers=127.14.227.148:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 03:16:36.435284 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:36.428358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:45991 (local address 127.14.227.190:34537)
0924 03:16:36.428849 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:36.428855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:36.430664 (+ 1809us) server_negotiation.cc:407] Connection header received
0924 03:16:36.432712 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:36.432729 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:36.432798 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:36.432918 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:36.434382 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:36.434390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:36.434397 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:36.434420 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:36.434473 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:36.434491 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:36.434618 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:36.434848 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 03:16:36.434930 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260924 03:16:36.436357 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:36.429022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34537 (local address 127.14.227.129:45991)
0924 03:16:36.430521 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:36.430569 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:36.432434 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:36.433142 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:36.433150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:36.433251 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:36.434046 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:16:36.434049 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:36.434106 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:16:36.434200 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:36.434601 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:36.434609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:36.434615 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:16:36.434931 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 03:16:36.435188 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":699,"threads_started":1}
I20260924 03:16:36.438074 25239 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34537
I20260924 03:16:36.438416 25239 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:36.439010 25239 heartbeater.cc:511] Master 127.14.227.190:34537 requested a full tablet report, sending...
I20260924 03:16:36.441241 25053 ts_manager.cc:194] Registered new tserver with Master: 0ca69f7ba3964e6e98c9ddee91dc061b (127.14.227.129:39681)
W20260924 03:16:36.759631 25243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:36.759893 25243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:36.759977 25243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:36.767136 25243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:36.767256 25243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:36.767403 25243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:16:36.776165 25243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:36.777338 25243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:36.779191 25243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:36.792052 25248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:36.792004 25249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:36.791999 25251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:36.792809 25243 server_base.cc:1034] running on GCE node
I20260924 03:16:36.793445 25243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:36.794432 25243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:36.795647 25243 hybrid_clock.cc:648] HybridClock initialized: now 1790219796795609 us; error 75 us; skew 500 ppm
I20260924 03:16:36.795953 25243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:16:36.798148 25243 webserver.cc:460] Webserver started at http://127.14.227.130:46575/ using document root <none> and password file <none>
I20260924 03:16:36.798831 25243 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:36.798960 25243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:36.799314 25243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:36.802438 25243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "580d3e47cf1b413abc676d7c36aef528"
format_stamp: "Formatted at 2026-09-24 03:16:36 on dist-test-slave-vd0p"
server_key: "af3e5d5a2a85c3263d64c5efabb11223"
server_key_iv: "d1530aceb40e4d48a6a1b22ad41d2172"
server_key_version: "encryptionkey@0"
I20260924 03:16:36.803216 25243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "580d3e47cf1b413abc676d7c36aef528"
format_stamp: "Formatted at 2026-09-24 03:16:36 on dist-test-slave-vd0p"
server_key: "af3e5d5a2a85c3263d64c5efabb11223"
server_key_iv: "d1530aceb40e4d48a6a1b22ad41d2172"
server_key_version: "encryptionkey@0"
I20260924 03:16:36.808152 25243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:16:36.811537 25257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:36.812734 25243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:16:36.812904 25243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "580d3e47cf1b413abc676d7c36aef528"
format_stamp: "Formatted at 2026-09-24 03:16:36 on dist-test-slave-vd0p"
server_key: "af3e5d5a2a85c3263d64c5efabb11223"
server_key_iv: "d1530aceb40e4d48a6a1b22ad41d2172"
server_key_version: "encryptionkey@0"
I20260924 03:16:36.813117 25243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:36.818763 25243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:36.819521 25243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:36.819785 25243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:36.820638 25243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:36.822106 25243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:36.822216 25243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:36.822362 25243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:36.822459 25243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:36.854904 25243 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:42255
I20260924 03:16:36.854987 25369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:42255 every 8 connection(s)
I20260924 03:16:36.856207 25243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:16:36.863826 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:36.858682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:44967 (local address 127.14.227.190:34537)
0924 03:16:36.858860 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:36.858865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:36.860236 (+ 1371us) server_negotiation.cc:407] Connection header received
0924 03:16:36.861833 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:36.861836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:36.861884 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:36.862027 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:36.863284 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:36.863287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:36.863290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:36.863303 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:36.863327 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:36.863338 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:36.863434 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:36.863584 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:16:36.863629 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:16:36.864702 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:36.859221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34537 (local address 127.14.227.130:44967)
0924 03:16:36.860069 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:36.860110 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:36.861597 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:36.862223 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:36.862231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:36.862334 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:36.863013 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:16:36.863015 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:36.863051 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:16:36.863131 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:36.863419 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:36.863426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:36.863431 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:16:36.863630 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 03:16:36.863814 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":256,"threads_started":1}
I20260924 03:16:36.865739 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 25243
I20260924 03:16:36.865931 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:16:36.866029 25370 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34537
I20260924 03:16:36.866328 25370 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:36.866492 15246 external_mini_cluster.cc:1428] Setting key 8514777000afe90c174eefc5819b3809
I20260924 03:16:36.866933 25370 heartbeater.cc:511] Master 127.14.227.190:34537 requested a full tablet report, sending...
I20260924 03:16:36.868366 25053 ts_manager.cc:194] Registered new tserver with Master: 580d3e47cf1b413abc676d7c36aef528 (127.14.227.130:42255)
I20260924 03:16:36.869299 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34537
--builtin_ntp_servers=127.14.227.148:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 03:16:37.182188 25373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:37.182488 25373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:37.182611 25373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:37.189508 25373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:37.189599 25373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:37.189814 25373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:16:37.198599 25373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:37.199838 25373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:37.201524 25373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:37.214222 25381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:37.214184 25379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:37.214277 25378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:37.214993 25373 server_base.cc:1034] running on GCE node
I20260924 03:16:37.215576 25373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:37.216517 25373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:37.217737 25373 hybrid_clock.cc:648] HybridClock initialized: now 1790219797217673 us; error 50 us; skew 500 ppm
I20260924 03:16:37.218099 25373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:16:37.220172 25373 webserver.cc:460] Webserver started at http://127.14.227.131:42331/ using document root <none> and password file <none>
I20260924 03:16:37.220817 25373 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:37.220947 25373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:37.221262 25373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:37.224082 25373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "ec4213689e45439f8a8ad56efa35dbc3"
format_stamp: "Formatted at 2026-09-24 03:16:37 on dist-test-slave-vd0p"
server_key: "389564aed6efab4c587ce50430cf39e4"
server_key_iv: "e119747d5c0069ee102a01dcc69be0a1"
server_key_version: "encryptionkey@0"
I20260924 03:16:37.224885 25373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "ec4213689e45439f8a8ad56efa35dbc3"
format_stamp: "Formatted at 2026-09-24 03:16:37 on dist-test-slave-vd0p"
server_key: "389564aed6efab4c587ce50430cf39e4"
server_key_iv: "e119747d5c0069ee102a01dcc69be0a1"
server_key_version: "encryptionkey@0"
I20260924 03:16:37.229773 25373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:16:37.233171 25387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:37.234467 25373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:16:37.234630 25373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "ec4213689e45439f8a8ad56efa35dbc3"
format_stamp: "Formatted at 2026-09-24 03:16:37 on dist-test-slave-vd0p"
server_key: "389564aed6efab4c587ce50430cf39e4"
server_key_iv: "e119747d5c0069ee102a01dcc69be0a1"
server_key_version: "encryptionkey@0"
I20260924 03:16:37.234827 25373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:37.240224 25373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:37.240942 25373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:37.241205 25373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:37.242141 25373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:37.243422 25373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:37.243554 25373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:37.243680 25373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:37.243772 25373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:37.274863 25373 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:44523
I20260924 03:16:37.274917 25499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:44523 every 8 connection(s)
I20260924 03:16:37.276134 25373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:16:37.279412 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 25373
I20260924 03:16:37.279606 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:16:37.280159 15246 external_mini_cluster.cc:1428] Setting key 12bf4e84fcc581667256cf2e1ae513ce
I20260924 03:16:37.284801 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:37.278704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:56265 (local address 127.14.227.190:34537)
0924 03:16:37.278893 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:37.278898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:37.280538 (+ 1640us) server_negotiation.cc:407] Connection header received
0924 03:16:37.282725 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:37.282730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:37.282795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:37.282906 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:37.284093 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:37.284099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:37.284103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:37.284120 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:37.284157 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:37.284168 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:37.284281 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:37.284472 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:16:37.284534 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:16:37.285641 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:37.279251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:34537 (local address 127.14.227.131:56265)
0924 03:16:37.280364 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:37.280413 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:37.282469 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:37.283131 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:37.283139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:37.283233 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:37.283819 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:16:37.283821 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:37.283863 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:16:37.283941 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:37.284264 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:37.284273 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:37.284281 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:16:37.284550 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 03:16:37.284782 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":320,"threads_started":1}
I20260924 03:16:37.286801 25500 heartbeater.cc:347] Connected to a master server at 127.14.227.190:34537
I20260924 03:16:37.287081 25500 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:37.287590 25500 heartbeater.cc:511] Master 127.14.227.190:34537 requested a full tablet report, sending...
I20260924 03:16:37.288769 25053 ts_manager.cc:194] Registered new tserver with Master: ec4213689e45439f8a8ad56efa35dbc3 (127.14.227.131:44523)
I20260924 03:16:37.295416 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 03:16:37.303987 25241 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:16:37.302999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.14.227.190:34537)
0924 03:16:37.303230 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:37.303238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:37.303255 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:16:37.303315 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:37.303318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:37.303359 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:37.303462 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:37.303788 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35558: BlockingRecv error: recv got EOF from 127.0.0.1:35558 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:16:37.304744 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 25112
I20260924 03:16:37.309512 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:37.305822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.14.227.129:39681)
0924 03:16:37.306385 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:37.306394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:37.306578 (+ 184us) server_negotiation.cc:407] Connection header received
0924 03:16:37.306653 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:37.306664 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:37.306947 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:37.307129 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:37.308505 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:37.308514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:37.308528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:37.308555 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:37.308612 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:37.308634 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:37.308784 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:37.309028 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 03:16:37.309138 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
I20260924 03:16:37.310988 25234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:37.395452 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 25112
I20260924 03:16:37.412038 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 25243
I20260924 03:16:37.415406 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:37.412885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.14.227.130:42255)
0924 03:16:37.413352 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:37.413359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:37.413383 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:16:37.413442 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:37.413448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:37.413662 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:37.413781 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:37.414704 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:37.414711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:37.414723 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:37.414734 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:37.414781 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:37.414796 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:37.414904 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:37.414989 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 03:16:37.415106 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":215,"threads_started":1}
I20260924 03:16:37.416673 25365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:37.499181 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 25243
I20260924 03:16:37.514737 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 25373
I20260924 03:16:37.518733 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:37.515796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.14.227.131:44523)
0924 03:16:37.516441 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:37.516448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:37.516477 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:16:37.516535 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:37.516541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:37.516778 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:37.516903 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:37.517914 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:37.517920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:37.517931 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:16:37.517942 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:37.518068 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:16:37.518083 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:37.518211 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:37.518320 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 03:16:37.518436 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":348,"threads_started":1}
I20260924 03:16:37.519934 25495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:37.599800 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 25373
I20260924 03:16:37.615125 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 25022
I20260924 03:16:37.616221 25083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:37.715484 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 25022
2026-09-24T03:16:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2287 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:16:37 dist-test-slave-vd0p krb5kdc[25518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:16:37 dist-test-slave-vd0p krb5kdc[25518](info): set up 2 sockets
Sep 24 03:16:37 dist-test-slave-vd0p krb5kdc[25518](info): commencing operation
krb5kdc: starting...
W20260924 03:16:38.806886 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:16:38 dist-test-slave-vd0p krb5kdc[25518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:16:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:16:39.020767 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43515
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:43515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:39.335395 25531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:39.335757 25531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:39.335920 25531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:39.343111 25531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:16:39.343255 25531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:39.343367 25531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:39.343478 25531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:16:39.343554 25531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:16:39.353170 25531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:43515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:43515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:39.354557 25531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:39.355824 25531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:39.367508 25537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:39.367568 25536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:39.367508 25539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:39.368361 25531 server_base.cc:1034] running on GCE node
I20260924 03:16:39.369029 25531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:39.370379 25531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:39.371598 25531 hybrid_clock.cc:648] HybridClock initialized: now 1790219799371555 us; error 33 us; skew 500 ppm
I20260924 03:16:39.371925 25531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:39 dist-test-slave-vd0p krb5kdc[25518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219799, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:39.374984 25531 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:16:39.376364 25531 webserver.cc:460] Webserver started at http://127.14.227.190:40927/ using document root <none> and password file <none>
I20260924 03:16:39.377018 25531 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:39.377162 25531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:39.377456 25531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:39.380332 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "8d2483b624a34e6f829b215b24558f6e"
format_stamp: "Formatted at 2026-09-24 03:16:39 on dist-test-slave-vd0p"
server_key: "66f7e069fbd3f6e8236708bbf0d09665"
server_key_iv: "cc2bd80a9c6902f87d4ae7ad5a7ac804"
server_key_version: "encryptionkey@0"
I20260924 03:16:39.381067 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "8d2483b624a34e6f829b215b24558f6e"
format_stamp: "Formatted at 2026-09-24 03:16:39 on dist-test-slave-vd0p"
server_key: "66f7e069fbd3f6e8236708bbf0d09665"
server_key_iv: "cc2bd80a9c6902f87d4ae7ad5a7ac804"
server_key_version: "encryptionkey@0"
I20260924 03:16:39.385895 25531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260924 03:16:39.390849 25546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:39.392884 25531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:16:39.393041 25531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "8d2483b624a34e6f829b215b24558f6e"
format_stamp: "Formatted at 2026-09-24 03:16:39 on dist-test-slave-vd0p"
server_key: "66f7e069fbd3f6e8236708bbf0d09665"
server_key_iv: "cc2bd80a9c6902f87d4ae7ad5a7ac804"
server_key_version: "encryptionkey@0"
I20260924 03:16:39.393208 25531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:39.412552 25531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:39.416328 25531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:39.416590 25531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:39.434396 25531 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:43515
I20260924 03:16:39.434412 25597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:43515 every 8 connection(s)
I20260924 03:16:39.435765 25531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:16:39.439528 25598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:39.440929 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 25531
I20260924 03:16:39.441180 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:16:39.441990 15246 external_mini_cluster.cc:1428] Setting key 4cddca43d1f9dcc2094d2291dafabc4f
I20260924 03:16:39.454162 25598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e: Bootstrap starting.
Sep 24 03:16:39 dist-test-slave-vd0p krb5kdc[25518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:16:39.457746 25598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:39.458878 25598 log.cc:824] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:39.461938 25598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e: No bootstrap required, opened a new log
I20260924 03:16:39.465278 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:39.444105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.14.227.190:43515)
0924 03:16:39.444896 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:39.444908 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:39.444951 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:16:39.446529 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:39.446554 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:39.446952 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:39.447274 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:39.449003 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:39.450352 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:39.452285 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:39.452957 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:39.453082 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:39.457535 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:39.457543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:39.457557 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:39.457595 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:39.461092 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:39.461804 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:39.461819 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:39.461826 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:39.462078 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:39.462566 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:39.462572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:39.462576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:39.463173 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:39.463422 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:39.463844 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 03:16:39.464243 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":389,"threads_started":1}
I20260924 03:16:39.471500 25598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2483b624a34e6f829b215b24558f6e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43515 } }
I20260924 03:16:39.471889 25598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:39.471993 25598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2483b624a34e6f829b215b24558f6e, State: Initialized, Role: FOLLOWER
I20260924 03:16:39.472522 25598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2483b624a34e6f829b215b24558f6e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43515 } }
I20260924 03:16:39.472757 25598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:16:39.472896 25598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:16:39.473070 25598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:39.475476 25598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2483b624a34e6f829b215b24558f6e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43515 } }
I20260924 03:16:39.475939 25598 leader_election.cc:302] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d2483b624a34e6f829b215b24558f6e; no voters:
I20260924 03:16:39.476490 25598 leader_election.cc:288] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:16:39.476603 25603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:39.477315 25603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [term 1 LEADER]: Becoming Leader. State: Replica: 8d2483b624a34e6f829b215b24558f6e, State: Running, Role: LEADER
I20260924 03:16:39.477843 25603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2483b624a34e6f829b215b24558f6e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43515 } }
I20260924 03:16:39.478618 25598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:16:39.480741 25604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d2483b624a34e6f829b215b24558f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2483b624a34e6f829b215b24558f6e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43515 } } }
I20260924 03:16:39.480825 25605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2483b624a34e6f829b215b24558f6e. Latest consensus state: current_term: 1 leader_uuid: "8d2483b624a34e6f829b215b24558f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2483b624a34e6f829b215b24558f6e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 43515 } } }
I20260924 03:16:39.481201 25604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:39.481230 25605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e [sys.catalog]: This master's current role is: LEADER
I20260924 03:16:39.482169 25609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:16:39.489837 25609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:16:39.498656 25609 catalog_manager.cc:1347] Generated new cluster ID: a3492344a28a484cb050854c8806aa0d
I20260924 03:16:39.498780 25609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:16:39.519163 25609 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:16:39.521076 25609 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:16:39.538796 25609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e: Generated new TSK 0
I20260924 03:16:39.539825 25609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:16:39.595103 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43515
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:39.912451 25624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:39.912719 25624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:39.912825 25624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:39.920110 25624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:39.920207 25624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:39.920307 25624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:16:39.929528 25624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:39.930711 25624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:39.931890 25624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:39.944649 25629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:39.944801 25630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:39.944969 25632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:39.945865 25624 server_base.cc:1034] running on GCE node
I20260924 03:16:39.946434 25624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:39.947312 25624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:39.948500 25624 hybrid_clock.cc:648] HybridClock initialized: now 1790219799948446 us; error 55 us; skew 500 ppm
I20260924 03:16:39.948783 25624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:39 dist-test-slave-vd0p krb5kdc[25518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219799, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:39.951730 25624 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:16:39.953063 25624 webserver.cc:460] Webserver started at http://127.14.227.129:40563/ using document root <none> and password file <none>
I20260924 03:16:39.953712 25624 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:39.953850 25624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:39.954172 25624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:39.957007 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "f1ec047edb7d4bcfacb24bc98eb67f55"
format_stamp: "Formatted at 2026-09-24 03:16:39 on dist-test-slave-vd0p"
server_key: "49866abcd4404805ec637e22874cf0fd"
server_key_iv: "c8b6e38d3f4d5a4b414e04aaaf708e23"
server_key_version: "encryptionkey@0"
I20260924 03:16:39.957764 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "f1ec047edb7d4bcfacb24bc98eb67f55"
format_stamp: "Formatted at 2026-09-24 03:16:39 on dist-test-slave-vd0p"
server_key: "49866abcd4404805ec637e22874cf0fd"
server_key_iv: "c8b6e38d3f4d5a4b414e04aaaf708e23"
server_key_version: "encryptionkey@0"
I20260924 03:16:39.962368 25624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:16:39.965778 25639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:39.967077 25624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:16:39.967257 25624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "f1ec047edb7d4bcfacb24bc98eb67f55"
format_stamp: "Formatted at 2026-09-24 03:16:39 on dist-test-slave-vd0p"
server_key: "49866abcd4404805ec637e22874cf0fd"
server_key_iv: "c8b6e38d3f4d5a4b414e04aaaf708e23"
server_key_version: "encryptionkey@0"
I20260924 03:16:39.967435 25624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:39.983513 25624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:39.987375 25624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:39.987670 25624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:39.988556 25624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:39.989807 25624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:39.989923 25624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:39.990098 25624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:39.990180 25624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:40.023250 25624 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:33819
I20260924 03:16:40.023284 25751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:33819 every 8 connection(s)
I20260924 03:16:40.024515 25624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:16:40.025462 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 25624
I20260924 03:16:40.025684 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:16:40.026500 15246 external_mini_cluster.cc:1428] Setting key 63ac4096fe6a622fc6495408ad66dad7
Sep 24 03:16:40 dist-test-slave-vd0p krb5kdc[25518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219799, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:16:40.046947 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:40.027502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:37613 (local address 127.14.227.190:43515)
0924 03:16:40.027961 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:40.027967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:40.029064 (+ 1097us) server_negotiation.cc:407] Connection header received
0924 03:16:40.031875 (+ 2811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:40.031879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:40.031952 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:40.032049 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:40.034172 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:40.034907 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:40.037276 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:40.037706 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:40.037843 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:40.041436 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:40.041441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:40.041444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:40.041474 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:40.043961 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:40.044670 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:40.044682 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:40.044686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:40.044807 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:40.045228 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:40.045233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:40.045236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:40.045484 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:40.045650 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:40.046352 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 03:16:40.046561 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260924 03:16:40.047904 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:40.028088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43515 (local address 127.14.227.129:37613)
0924 03:16:40.028875 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:40.028937 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:40.031595 (+ 2658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:40.032314 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:40.032330 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:40.033091 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:40.033934 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:40.033998 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:40.035065 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:40.035073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:40.037129 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:40.037141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:40.037939 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:40.037967 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:40.038205 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:40.039021 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:40.039057 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:40.041181 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:40.044104 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:40.044113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:40.044130 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:40.044545 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:40.044914 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:40.044919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:40.044922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:40.045110 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:40.045610 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:40.045619 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:40.045995 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 03:16:40.046262 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 03:16:40.046790 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":323,"threads_started":1}
I20260924 03:16:40.049549 25752 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43515
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:16:40.049857 25752 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:40.050606 25752 heartbeater.cc:511] Master 127.14.227.190:43515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:16:40.052760 25563 ts_manager.cc:194] Registered new tserver with Master: f1ec047edb7d4bcfacb24bc98eb67f55 (127.14.227.129:33819)
I20260924 03:16:40.054234 25563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:37613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:16:40.081266 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43515
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:40.395056 25758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:40.395325 25758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:40.395433 25758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:40.402920 25758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:40.403028 25758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:40.403144 25758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:16:40.412588 25758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:40.413769 25758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:40.415064 25758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:40.428876 25764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:40.428812 25763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:40.428845 25766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:40.429729 25758 server_base.cc:1034] running on GCE node
I20260924 03:16:40.430357 25758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:40.431268 25758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:40.432485 25758 hybrid_clock.cc:648] HybridClock initialized: now 1790219800432451 us; error 47 us; skew 500 ppm
I20260924 03:16:40.432789 25758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:40 dist-test-slave-vd0p krb5kdc[25518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219800, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:40.441854 25758 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:16:40.443398 25758 webserver.cc:460] Webserver started at http://127.14.227.130:43195/ using document root <none> and password file <none>
I20260924 03:16:40.444103 25758 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:40.444228 25758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:40.444530 25758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:40.447429 25758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "c5a2ab9b298841fc90eb75c184ad7fe3"
format_stamp: "Formatted at 2026-09-24 03:16:40 on dist-test-slave-vd0p"
server_key: "d8148cd2837cf66b971232d6afda1243"
server_key_iv: "4a872e4a05c27041614477e73d036e2c"
server_key_version: "encryptionkey@0"
I20260924 03:16:40.448174 25758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "c5a2ab9b298841fc90eb75c184ad7fe3"
format_stamp: "Formatted at 2026-09-24 03:16:40 on dist-test-slave-vd0p"
server_key: "d8148cd2837cf66b971232d6afda1243"
server_key_iv: "4a872e4a05c27041614477e73d036e2c"
server_key_version: "encryptionkey@0"
I20260924 03:16:40.453053 25758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:16:40.456369 25773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:40.457711 25758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:16:40.457834 25758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "c5a2ab9b298841fc90eb75c184ad7fe3"
format_stamp: "Formatted at 2026-09-24 03:16:40 on dist-test-slave-vd0p"
server_key: "d8148cd2837cf66b971232d6afda1243"
server_key_iv: "4a872e4a05c27041614477e73d036e2c"
server_key_version: "encryptionkey@0"
I20260924 03:16:40.458058 25758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:40.487883 25758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:40.492393 25758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:40.492702 25758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:40.493709 25758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:40.495090 25758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:40.495162 25758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:40.495285 25758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:40.495363 25758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:40.530057 25758 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:45655
I20260924 03:16:40.530174 25885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:45655 every 8 connection(s)
I20260924 03:16:40.531368 25758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:16:40.541683 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 25758
I20260924 03:16:40.541884 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:16:40.542330 15246 external_mini_cluster.cc:1428] Setting key f23ea6f8a956dc41bd3818fc85f03869
Sep 24 03:16:40 dist-test-slave-vd0p krb5kdc[25518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219800, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:16:40.551183 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:40.533992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:57105 (local address 127.14.227.190:43515)
0924 03:16:40.534159 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:40.534163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:40.535252 (+ 1089us) server_negotiation.cc:407] Connection header received
0924 03:16:40.536982 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:40.536988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:40.537077 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:40.537193 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:40.538685 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:40.539460 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:40.540971 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:40.541375 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:40.541510 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:40.545252 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:40.545263 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:40.545285 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:40.545332 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:40.548052 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:40.548959 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:40.548963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:40.548965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:40.549089 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:40.549568 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:40.549571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:40.549573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:40.549809 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:40.549913 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:40.550791 (+ 878us) server_negotiation.cc:299] Negotiation successful
0924 03:16:40.550953 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:16:40.552343 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:40.534440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43515 (local address 127.14.227.130:57105)
0924 03:16:40.535089 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:40.535134 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:40.536715 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:40.537426 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:40.537434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:40.537917 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:40.538522 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:40.538539 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:40.539609 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:40.539616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:40.540828 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:40.540840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:40.541591 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:40.541597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:40.541857 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:40.542580 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:40.542613 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:40.544940 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:40.548196 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:40.548207 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:40.548227 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:40.548836 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:40.549193 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:40.549200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:40.549206 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:40.549451 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:40.549930 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:40.549941 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:40.550400 (+ 459us) client_negotiation.cc:769] Sending connection context
0924 03:16:40.550705 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 03:16:40.551282 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":258,"threads_started":1}
I20260924 03:16:40.553730 25886 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43515
I20260924 03:16:40.554095 25886 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:40.554796 25886 heartbeater.cc:511] Master 127.14.227.190:43515 requested a full tablet report, sending...
I20260924 03:16:40.556237 25563 ts_manager.cc:194] Registered new tserver with Master: c5a2ab9b298841fc90eb75c184ad7fe3 (127.14.227.130:45655)
I20260924 03:16:40.557042 25563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:57105
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:16:40.592700 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43515
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:16:40.905599 25891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:16:40.905850 25891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:16:40.906013 25891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:16:40.912973 25891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:16:40.913065 25891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:16:40.913288 25891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:16:40.922570 25891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:43515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:16:40.923817 25891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:16:40.925078 25891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:16:40.937634 25896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:40.937670 25899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:16:40.937716 25897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:16:40.939179 25891 server_base.cc:1034] running on GCE node
I20260924 03:16:40.939795 25891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:16:40.940631 25891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:16:40.941826 25891 hybrid_clock.cc:648] HybridClock initialized: now 1790219800941742 us; error 85 us; skew 500 ppm
I20260924 03:16:40.942154 25891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:16:40 dist-test-slave-vd0p krb5kdc[25518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219800, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:16:40.945147 25891 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:16:40.946604 25891 webserver.cc:460] Webserver started at http://127.14.227.131:46599/ using document root <none> and password file <none>
I20260924 03:16:40.947254 25891 fs_manager.cc:359] Metadata directory not provided
I20260924 03:16:40.947405 25891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:16:40.947696 25891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:16:40.950461 25891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "7f704f5f590a4714a49d1606b9f5a77a"
format_stamp: "Formatted at 2026-09-24 03:16:40 on dist-test-slave-vd0p"
server_key: "acf11a2a8021f03ddd7b56fd6baccad0"
server_key_iv: "482de922fc88d0877bbb7d47be071038"
server_key_version: "encryptionkey@0"
I20260924 03:16:40.951215 25891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "7f704f5f590a4714a49d1606b9f5a77a"
format_stamp: "Formatted at 2026-09-24 03:16:40 on dist-test-slave-vd0p"
server_key: "acf11a2a8021f03ddd7b56fd6baccad0"
server_key_iv: "482de922fc88d0877bbb7d47be071038"
server_key_version: "encryptionkey@0"
I20260924 03:16:40.955902 25891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 03:16:40.959100 25906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:40.960358 25891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:16:40.960528 25891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "7f704f5f590a4714a49d1606b9f5a77a"
format_stamp: "Formatted at 2026-09-24 03:16:40 on dist-test-slave-vd0p"
server_key: "acf11a2a8021f03ddd7b56fd6baccad0"
server_key_iv: "482de922fc88d0877bbb7d47be071038"
server_key_version: "encryptionkey@0"
I20260924 03:16:40.960737 25891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:16:40.977840 25891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:16:40.982086 25891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:16:40.982363 25891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:16:40.983254 25891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:16:40.984522 25891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:16:40.984633 25891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:40.984782 25891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:16:40.984855 25891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:16:41.017174 25891 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:44223
I20260924 03:16:41.017261 26018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:44223 every 8 connection(s)
I20260924 03:16:41.018472 25891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:16:41.023702 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 25891
I20260924 03:16:41.023888 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:16:41.024405 15246 external_mini_cluster.cc:1428] Setting key 86db3000aa0bda17f7517cd74186e0fa
Sep 24 03:16:41 dist-test-slave-vd0p krb5kdc[25518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219800, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:16:41.038554 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.021027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:33103 (local address 127.14.227.190:43515)
0924 03:16:41.021201 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.021206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.022385 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 03:16:41.024686 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.024692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.024787 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.024894 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:41.026817 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.027985 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.029417 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.029834 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.029926 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.033191 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:41.033196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:41.033199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:41.033226 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:41.035703 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.036414 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.036421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.036429 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.036540 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.036930 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.036935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.036937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.037186 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:41.037332 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.037991 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.038204 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:16:41.039556 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.021511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:43515 (local address 127.14.227.131:33103)
0924 03:16:41.022237 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:41.022274 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:41.024357 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:41.025169 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:41.025178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:41.025876 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:16:41.026650 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.026678 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.028128 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.028132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.029295 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.029306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.030062 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.030067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.030256 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.030836 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:16:41.030861 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:16:41.032999 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:16:41.035839 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:41.035848 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:41.035862 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:41.036279 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:41.036643 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:16:41.036648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:16:41.036652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:16:41.036819 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:16:41.037306 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:16:41.037313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:16:41.037637 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 03:16:41.037879 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 03:16:41.038368 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":253,"threads_started":1}
I20260924 03:16:41.040796 26019 heartbeater.cc:347] Connected to a master server at 127.14.227.190:43515
I20260924 03:16:41.041064 26019 heartbeater.cc:464] Registering TS with master...
I20260924 03:16:41.041692 26019 heartbeater.cc:511] Master 127.14.227.190:43515 requested a full tablet report, sending...
I20260924 03:16:41.042973 25563 ts_manager.cc:194] Registered new tserver with Master: 7f704f5f590a4714a49d1606b9f5a77a (127.14.227.131:44223)
I20260924 03:16:41.043686 25563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:33103
I20260924 03:16:41.051549 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:16:41.057475 25752 heartbeater.cc:503] Master 127.14.227.190:43515 was elected leader, sending a full tablet report...
I20260924 03:16:41.067662 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.055698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.14.227.190:43515)
0924 03:16:41.055851 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.055855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.056325 (+ 470us) server_negotiation.cc:407] Connection header received
0924 03:16:41.056445 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.056448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.056499 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.056565 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:41.058161 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.059162 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.060315 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.060704 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.060816 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.063198 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:41.063202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:41.063206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:41.063231 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:41.065592 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.066197 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.066200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.066202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.066341 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.066654 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.066657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.066659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.066913 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:41.067079 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.067298 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.067441 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:16:41.071237 25563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:16:41.077711 25563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:16:41.103839 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.088485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:33819 (local address 127.0.0.1:55520)
0924 03:16:41.094406 (+ 5921us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:41.094482 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:41.094730 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:41.095680 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:41.095687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:41.095723 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:41.096184 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.096216 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.098233 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.098238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.100626 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.100638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.102824 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.102830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.103031 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.103082 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 03:16:41.103205 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 03:16:41.103381 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5649,"mutex_wait_us":48,"spinlock_wait_cycles":8448,"thread_start_us":5497,"threads_started":1}
I20260924 03:16:41.104753 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.088803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.14.227.129:33819)
0924 03:16:41.089389 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.089398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.094575 (+ 5177us) server_negotiation.cc:407] Connection header received
0924 03:16:41.094952 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.094960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.095200 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.095357 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:41.096449 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.098066 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.100808 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.102292 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.102496 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.102705 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.104240 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.104415 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":846,"threads_started":1}
I20260924 03:16:41.106324 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.089387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.14.227.131:44223)
0924 03:16:41.093997 (+ 4610us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.094008 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.094575 (+ 567us) server_negotiation.cc:407] Connection header received
0924 03:16:41.094956 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.094962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.095200 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.095356 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:41.097449 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.099095 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.102593 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.104021 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.104167 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.104320 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.105827 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.106012 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4410,"thread_start_us":4277,"threads_started":1}
I20260924 03:16:41.107148 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.088740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:44223 (local address 127.0.0.1:53482)
0924 03:16:41.094406 (+ 5666us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:41.094482 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:41.094729 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:41.095680 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:41.095687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:41.095725 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:41.097217 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.097229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.099266 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.099272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.102414 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.102435 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.104500 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.104506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.104682 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.104712 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:16:41.106524 (+ 1812us) client_negotiation.cc:240] Negotiation successful
0924 03:16:41.106691 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5530,"mutex_wait_us":2,"spinlock_wait_cycles":12160,"thread_start_us":5382,"threads_started":1}
I20260924 03:16:41.109572 25687 tablet_service.cc:1467] Processing CreateTablet for tablet 1173bd4746cc4b7198f2158a0dc08d70 (DEFAULT_TABLE table=test-table [id=854ab249de6d4c8bb51b10fde168d8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:41.111158 25687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1173bd4746cc4b7198f2158a0dc08d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:41.110884 25954 tablet_service.cc:1467] Processing CreateTablet for tablet 1173bd4746cc4b7198f2158a0dc08d70 (DEFAULT_TABLE table=test-table [id=854ab249de6d4c8bb51b10fde168d8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:41.112344 25954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1173bd4746cc4b7198f2158a0dc08d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:41.112982 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.094837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:45655 (local address 127.0.0.1:39984)
0924 03:16:41.105144 (+ 10307us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:41.105163 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:41.105250 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:41.108003 (+ 2753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:41.108008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:41.108033 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:41.108177 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.108187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.109400 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.109408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.110734 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.110747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.112240 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.112246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.112414 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.112468 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 03:16:41.112566 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:16:41.112702 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10213,"thread_start_us":12097,"threads_started":1}
I20260924 03:16:41.113220 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.095140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.14.227.130:45655)
0924 03:16:41.107483 (+ 12343us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.107495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.107526 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:16:41.107645 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.107651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.107828 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.108021 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:41.108318 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.109267 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.110912 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.112081 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.112254 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.112410 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.112620 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.112765 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12168,"thread_start_us":12045,"threads_started":1}
I20260924 03:16:41.115906 25821 tablet_service.cc:1467] Processing CreateTablet for tablet 1173bd4746cc4b7198f2158a0dc08d70 (DEFAULT_TABLE table=test-table [id=854ab249de6d4c8bb51b10fde168d8c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:16:41.116886 25821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1173bd4746cc4b7198f2158a0dc08d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:16:41.125660 26037 tablet_bootstrap.cc:492] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Bootstrap starting.
I20260924 03:16:41.127936 26039 tablet_bootstrap.cc:492] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Bootstrap starting.
I20260924 03:16:41.129983 26037 tablet_bootstrap.cc:654] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:41.130663 26038 tablet_bootstrap.cc:492] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Bootstrap starting.
I20260924 03:16:41.131388 26037 log.cc:824] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:41.131834 26039 tablet_bootstrap.cc:654] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:41.133066 26039 log.cc:824] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:41.133862 26038 tablet_bootstrap.cc:654] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Neither blocks nor log segments found. Creating new log.
I20260924 03:16:41.134657 26037 tablet_bootstrap.cc:492] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: No bootstrap required, opened a new log
I20260924 03:16:41.134907 26037 ts_tablet_manager.cc:1397] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260924 03:16:41.134935 26038 log.cc:824] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:16:41.135473 26039 tablet_bootstrap.cc:492] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: No bootstrap required, opened a new log
I20260924 03:16:41.135756 26039 ts_tablet_manager.cc:1397] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260924 03:16:41.137589 26038 tablet_bootstrap.cc:492] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: No bootstrap required, opened a new log
I20260924 03:16:41.137830 26038 ts_tablet_manager.cc:1397] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 03:16:41.142989 26039 raft_consensus.cc:348] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.143350 26039 raft_consensus.cc:374] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:41.143453 26039 raft_consensus.cc:729] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a2ab9b298841fc90eb75c184ad7fe3, State: Initialized, Role: FOLLOWER
I20260924 03:16:41.144147 26039 consensus_queue.cc:260] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.145236 25886 heartbeater.cc:503] Master 127.14.227.190:43515 was elected leader, sending a full tablet report...
I20260924 03:16:41.145761 26039 ts_tablet_manager.cc:1428] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260924 03:16:41.146660 26037 raft_consensus.cc:348] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.147038 26037 raft_consensus.cc:374] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:41.147184 26037 raft_consensus.cc:729] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ec047edb7d4bcfacb24bc98eb67f55, State: Initialized, Role: FOLLOWER
I20260924 03:16:41.147781 26037 consensus_queue.cc:260] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.148092 26038 raft_consensus.cc:348] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.148546 26038 raft_consensus.cc:374] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:16:41.148689 26038 raft_consensus.cc:729] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f704f5f590a4714a49d1606b9f5a77a, State: Initialized, Role: FOLLOWER
I20260924 03:16:41.149420 26037 ts_tablet_manager.cc:1428] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260924 03:16:41.149387 26038 consensus_queue.cc:260] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.151278 26019 heartbeater.cc:503] Master 127.14.227.190:43515 was elected leader, sending a full tablet report...
I20260924 03:16:41.151885 26038 ts_tablet_manager.cc:1428] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
W20260924 03:16:41.270133 26020 tablet.cc:2369] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:16:41.276983 25753 tablet.cc:2369] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:16:41.283252 25887 tablet.cc:2369] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:16:41.510340 26043 raft_consensus.cc:482] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:16:41.510692 26043 raft_consensus.cc:504] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.512262 26043 leader_election.cc:288] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1ec047edb7d4bcfacb24bc98eb67f55 (127.14.227.129:33819), 7f704f5f590a4714a49d1606b9f5a77a (127.14.227.131:44223)
I20260924 03:16:41.521411 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.512819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:43765 (local address 127.14.227.129:33819)
0924 03:16:41.513021 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.513027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.513811 (+ 784us) server_negotiation.cc:407] Connection header received
0924 03:16:41.514016 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.514022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.514128 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.514220 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:41.514973 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.516102 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.519059 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.520299 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.520457 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.520532 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.521014 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.521172 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:16:41.521410 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.512842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:43481 (local address 127.14.227.131:44223)
0924 03:16:41.513021 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.513027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.513811 (+ 784us) server_negotiation.cc:407] Connection header received
0924 03:16:41.514016 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.514022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.514116 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.514220 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:16:41.514973 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.516104 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.519059 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.520294 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.520457 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.520531 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.521014 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.521151 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:16:41.521699 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.512662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:33819 (local address 127.14.227.130:43765)
0924 03:16:41.513690 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:41.513705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:41.513851 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:41.514251 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:41.514257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:41.514282 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:41.514807 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.514826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.516237 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.516241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.518894 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.518909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.520563 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.520570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.520813 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.520877 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 03:16:41.520971 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 03:16:41.521164 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":284,"thread_start_us":594,"threads_started":1}
I20260924 03:16:41.521724 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.512561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:44223 (local address 127.14.227.130:43481)
0924 03:16:41.513643 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0924 03:16:41.513664 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:16:41.513867 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:16:41.514251 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:16:41.514257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:16:41.514292 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:16:41.514830 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.514838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.516234 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.516239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.518894 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:16:41.518909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.520565 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.520570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:16:41.520810 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.520872 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 03:16:41.520971 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 03:16:41.521153 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":210,"spinlock_wait_cycles":67328,"thread_start_us":705,"threads_started":1}
I20260924 03:16:41.523079 25974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1173bd4746cc4b7198f2158a0dc08d70" candidate_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f704f5f590a4714a49d1606b9f5a77a" is_pre_election: true
I20260924 03:16:41.523079 25707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1173bd4746cc4b7198f2158a0dc08d70" candidate_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" is_pre_election: true
I20260924 03:16:41.523653 25974 raft_consensus.cc:2454] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a2ab9b298841fc90eb75c184ad7fe3 in term 0.
I20260924 03:16:41.523643 25707 raft_consensus.cc:2454] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a2ab9b298841fc90eb75c184ad7fe3 in term 0.
I20260924 03:16:41.524348 25777 leader_election.cc:302] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7f704f5f590a4714a49d1606b9f5a77a, c5a2ab9b298841fc90eb75c184ad7fe3, f1ec047edb7d4bcfacb24bc98eb67f55; no voters:
I20260924 03:16:41.524812 26043 raft_consensus.cc:2781] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:16:41.524968 26043 raft_consensus.cc:482] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:16:41.525075 26043 raft_consensus.cc:3037] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:41.527284 26043 raft_consensus.cc:504] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.527928 26043 leader_election.cc:288] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [CANDIDATE]: Term 1 election: Requested vote from peers f1ec047edb7d4bcfacb24bc98eb67f55 (127.14.227.129:33819), 7f704f5f590a4714a49d1606b9f5a77a (127.14.227.131:44223)
I20260924 03:16:41.528352 25707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1173bd4746cc4b7198f2158a0dc08d70" candidate_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55"
I20260924 03:16:41.528379 25974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1173bd4746cc4b7198f2158a0dc08d70" candidate_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f704f5f590a4714a49d1606b9f5a77a"
I20260924 03:16:41.528556 25974 raft_consensus.cc:3037] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:41.528556 25707 raft_consensus.cc:3037] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:16:41.531675 25974 raft_consensus.cc:2454] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a2ab9b298841fc90eb75c184ad7fe3 in term 1.
I20260924 03:16:41.531675 25707 raft_consensus.cc:2454] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a2ab9b298841fc90eb75c184ad7fe3 in term 1.
I20260924 03:16:41.532068 25776 leader_election.cc:302] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f704f5f590a4714a49d1606b9f5a77a, c5a2ab9b298841fc90eb75c184ad7fe3; no voters:
I20260924 03:16:41.532392 26043 raft_consensus.cc:2781] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:16:41.532895 26043 raft_consensus.cc:686] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 1 LEADER]: Becoming Leader. State: Replica: c5a2ab9b298841fc90eb75c184ad7fe3, State: Running, Role: LEADER
I20260924 03:16:41.533416 26043 consensus_queue.cc:237] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } }
I20260924 03:16:41.538436 25563 catalog_manager.cc:5495] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5a2ab9b298841fc90eb75c184ad7fe3 (127.14.227.130). New cstate: current_term: 1 leader_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5a2ab9b298841fc90eb75c184ad7fe3" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 45655 } health_report { overall_health: HEALTHY } } }
I20260924 03:16:41.659960 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.654905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.14.227.130:45655)
0924 03:16:41.655540 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.655544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.655559 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:16:41.655640 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.655643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.655717 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.655842 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:16:41.656208 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.656833 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.658182 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.658552 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.658646 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.659013 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:16:41.659526 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:16:41.659631 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.659702 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.659760 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":404,"threads_started":1}
I20260924 03:16:41.667492 25974 raft_consensus.cc:1261] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 1 FOLLOWER]: Refusing update from remote peer c5a2ab9b298841fc90eb75c184ad7fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:41.667546 25707 raft_consensus.cc:1261] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 1 FOLLOWER]: Refusing update from remote peer c5a2ab9b298841fc90eb75c184ad7fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:16:41.668361 26048 consensus_queue.cc:1035] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f704f5f590a4714a49d1606b9f5a77a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 44223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:41.668722 26043 consensus_queue.cc:1035] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ec047edb7d4bcfacb24bc98eb67f55" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:16:41.679024 26053 mvcc.cc:205] Tried to move back new op lower bound from 7332740307621076992 to 7332740307085213696. Current Snapshot: MvccSnapshot[applied={T|T < 7332740307621076992}]
I20260924 03:16:41.684029 26056 mvcc.cc:205] Tried to move back new op lower bound from 7332740307621076992 to 7332740307085213696. Current Snapshot: MvccSnapshot[applied={T|T < 7332740307621076992 or (T in {7332740307621076992})}]
I20260924 03:16:41.689459 26058 mvcc.cc:205] Tried to move back new op lower bound from 7332740307621076992 to 7332740307085213696. Current Snapshot: MvccSnapshot[applied={T|T < 7332740307621076992}]
I20260924 03:16:41.693054 25563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:41.693341 25563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:16:41.697312 25563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d2483b624a34e6f829b215b24558f6e: Sending DeleteTablet for 3 replicas of tablet 1173bd4746cc4b7198f2158a0dc08d70
I20260924 03:16:41.698478 25687 tablet_service.cc:1514] Processing DeleteTablet for tablet 1173bd4746cc4b7198f2158a0dc08d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:41 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:55520
I20260924 03:16:41.698575 25821 tablet_service.cc:1514] Processing DeleteTablet for tablet 1173bd4746cc4b7198f2158a0dc08d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:41 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:39984
I20260924 03:16:41.698737 25954 tablet_service.cc:1514] Processing DeleteTablet for tablet 1173bd4746cc4b7198f2158a0dc08d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:41 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:53482
I20260924 03:16:41.699254 25953 tablet_service.cc:1514] Processing DeleteTablet for tablet 1173bd4746cc4b7198f2158a0dc08d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:16:41 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:53482
I20260924 03:16:41.699415 26065 tablet_replica.cc:330] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: stopping tablet replica
I20260924 03:16:41.699833 26065 raft_consensus.cc:2229] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:41.700356 26065 raft_consensus.cc:2258] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:41.700505 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 25624
I20260924 03:16:41.702284 26065 ts_tablet_manager.cc:1905] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:41.709523 26067 tablet_replica.cc:330] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: stopping tablet replica
I20260924 03:16:41.710071 26067 raft_consensus.cc:2229] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:16:41.710696 26066 tablet_replica.cc:330] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: stopping tablet replica
I20260924 03:16:41.710762 26065 ts_tablet_manager.cc:1918] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:41.710961 26065 log.cc:1196] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/1173bd4746cc4b7198f2158a0dc08d70
I20260924 03:16:41.711205 26067 raft_consensus.cc:2258] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:41.711483 26065 ts_tablet_manager.cc:1939] T 1173bd4746cc4b7198f2158a0dc08d70 P f1ec047edb7d4bcfacb24bc98eb67f55: Deleting consensus metadata
I20260924 03:16:41.711997 26066 raft_consensus.cc:2229] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:16:41.713395 25550 catalog_manager.cc:4841] TS f1ec047edb7d4bcfacb24bc98eb67f55 (127.14.227.129:33819): tablet 1173bd4746cc4b7198f2158a0dc08d70 (table test-table [id=854ab249de6d4c8bb51b10fde168d8c9]) successfully deleted
I20260924 03:16:41.714165 26066 raft_consensus.cc:2258] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:16:41.714668 26067 ts_tablet_manager.cc:1905] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:16:41.715951 26066 ts_tablet_manager.cc:1905] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:16:41 dist-test-slave-vd0p krb5kdc[25518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:16:41.722918 26067 ts_tablet_manager.cc:1918] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:41.723083 26067 log.cc:1196] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/1173bd4746cc4b7198f2158a0dc08d70
I20260924 03:16:41.723604 26067 ts_tablet_manager.cc:1939] T 1173bd4746cc4b7198f2158a0dc08d70 P 7f704f5f590a4714a49d1606b9f5a77a: Deleting consensus metadata
I20260924 03:16:41.725394 25549 catalog_manager.cc:4841] TS 7f704f5f590a4714a49d1606b9f5a77a (127.14.227.131:44223): tablet 1173bd4746cc4b7198f2158a0dc08d70 (table test-table [id=854ab249de6d4c8bb51b10fde168d8c9]) successfully deleted
I20260924 03:16:41.725569 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.702967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.14.227.129:33819)
0924 03:16:41.703155 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.703161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.710444 (+ 7283us) server_negotiation.cc:407] Connection header received
0924 03:16:41.710682 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.710687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.710781 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.710891 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:41.712706 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.713794 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.715369 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.715770 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.715910 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.719776 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:41.719786 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:41.719799 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:41.719880 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:41.722729 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.723496 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.723503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.723505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.723646 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.724042 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.724047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.724050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.724366 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:41.724721 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.724979 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.725187 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:16:41.726337 25747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:41.733480 26066 ts_tablet_manager.cc:1918] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:16:41.733659 26066 log.cc:1196] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/1173bd4746cc4b7198f2158a0dc08d70
I20260924 03:16:41.734261 26066 ts_tablet_manager.cc:1939] T 1173bd4746cc4b7198f2158a0dc08d70 P c5a2ab9b298841fc90eb75c184ad7fe3: Deleting consensus metadata
W20260924 03:16:41.734938 25549 catalog_manager.cc:4808] TS 7f704f5f590a4714a49d1606b9f5a77a (127.14.227.131:44223): delete failed for tablet 1173bd4746cc4b7198f2158a0dc08d70 because the tablet was not found. No further retry: Not found: Tablet not found: 1173bd4746cc4b7198f2158a0dc08d70
I20260924 03:16:41.736220 25549 catalog_manager.cc:4841] TS c5a2ab9b298841fc90eb75c184ad7fe3 (127.14.227.130:45655): tablet 1173bd4746cc4b7198f2158a0dc08d70 (table test-table [id=854ab249de6d4c8bb51b10fde168d8c9]) successfully deleted
I20260924 03:16:41.815644 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 25624
I20260924 03:16:41.837793 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 25758
Sep 24 03:16:41 dist-test-slave-vd0p krb5kdc[25518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:16:41.854557 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.838616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.14.227.130:45655)
0924 03:16:41.838815 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.838822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.838849 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:16:41.839040 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.839046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.839135 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.839251 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:41.840484 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.841661 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.843041 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.843714 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.843849 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.847793 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:41.847808 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:41.847830 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:41.847872 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:41.851788 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.852463 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.852472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.852475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.852624 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.852943 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.852948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.852952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.853400 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:41.853708 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.853891 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.854171 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:16:41.855314 25881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:41.964285 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 25758
I20260924 03:16:41.982091 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 25891
Sep 24 03:16:41 dist-test-slave-vd0p krb5kdc[25518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:16:41.996675 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:16:41.982791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.14.227.131:44223)
0924 03:16:41.982965 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:16:41.982972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:16:41.983034 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:16:41.983234 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:16:41.983241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:16:41.983326 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:16:41.983429 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:16:41.984659 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.985351 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.986661 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:16:41.987056 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:16:41.987190 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:16:41.991416 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:16:41.991431 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:16:41.991446 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:16:41.991477 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:16:41.994054 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.994696 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.994703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.994705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.994832 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:16:41.995194 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:16:41.995201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:16:41.995204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:16:41.995544 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:16:41.995857 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 03:16:41.996115 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 03:16:41.996328 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:16:41.997457 26014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:42.084148 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 25891
I20260924 03:16:42.100538 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 25531
I20260924 03:16:42.101728 25593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:16:42.207684 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 25531
2026-09-24T03:16:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4493 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:16:42 dist-test-slave-vd0p krb5kdc[26075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:16:42 dist-test-slave-vd0p krb5kdc[26075](info): set up 2 sockets
Sep 24 03:16:42 dist-test-slave-vd0p krb5kdc[26075](info): commencing operation
krb5kdc: starting...
W20260924 03:16:43.286851 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:16:43 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:16:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/rangeradmin_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/rangeradmin_127.14.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/rangerlookup_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/rangerlookup_127.14.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/HTTP_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/HTTP_127.14.227.148@KRBTEST.COM.keytab.
I20260924 03:16:43.601007 15246 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1C3avF/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/postgres -l logfile start
2026-09-24 03:16:46.044 UTC [26105] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:16:46.044 UTC [26105] LOG: listening on IPv4 address "127.14.227.148", port 57773
2026-09-24 03:16:46.047 UTC [26105] LOG: listening on Unix socket "/tmp/.s.PGSQL.57773"
2026-09-24 03:16:46.069 UTC [26108] LOG: database system was shut down at 2026-09-24 03:16:44 UTC
2026-09-24 03:16:46.072 UTC [26105] LOG: database system is ready to accept connections
I20260924 03:16:47.034699 15246 mini_postgres.cc:96] Postgres bound to 57773
2026-09-24 03:16:47.052 UTC [26116] FATAL: database "slave" does not exist
127.14.227.148:57773 - accepting connections
I20260924 03:16:47.052892 15246 mini_ranger.cc:161] Starting Ranger...
I20260924 03:16:47.077672 15246 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:16:47.566866 15246 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:16:47.566991 15246 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-admin
2026-09-24 03:16:47,840 [I] DB FLAVOR :POSTGRES
2026-09-24 03:16:47,840 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:16:47,840 [I] Checking connection..
2026-09-24 03:16:47,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:16:48,074 [I] Checking connection passed.
2026-09-24 03:16:48,074 [I] --------- Verifying version history table ---------
2026-09-24 03:16:48,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:16:48,332 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:16:48,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:16:48,587 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:16:48,587 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:16:48,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:16:48.826 UTC [26180] WARNING: there is no transaction in progress
2026-09-24 03:16:48,832 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:16:48,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:16:49,083 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:16:49,084 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:16:49,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:16:49,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:16:49,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vd0p.c.gcp-upstream.internal','N') ;"
2026-09-24 03:16:49,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:16:50,052 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:16:50,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:16:50,308 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:16:50,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:16:50,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:16:50,756 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:16:50,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:16:51.699 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.715 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.731 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.746 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.761 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.778 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.841 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.848 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.862 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.872 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.882 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.896 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.905 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.911 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.917 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:51.923 UTC [26288] WARNING: there is no transaction in progress
2026-09-24 03:16:52,532 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:16:52,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:16:52,778 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:16:52,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:16:53,005 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:16:53,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:16:53,261 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:16:53,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:16:53,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:16:53,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vd0p.c.gcp-upstream.internal';"
2026-09-24 03:16:54,004 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:16:54,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:16:54,255 [I] DB_PATCHES have already been applied
I20260924 03:16:54.261224 15246 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-admin:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1C3avF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:16:54.261304 15246 mini_ranger.cc:193] Using host: 127.14.227.148
I20260924 03:16:54.275082 15246 mini_ranger.cc:227] Ranger admin URL: http://127.14.227.148:47091
Sep 24, 2026 3:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:16:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:16:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:16:55.460247 15246 mini_ranger.cc:160] Time spent starting Ranger: real 8.407s user 0.001s sys 0.034s
Sep 24, 2026 3:17:15 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:17:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 3:17:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:17:15 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:17:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:17:16.791332 15246 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/rangerkms_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/rangerkms_127.14.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.227.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.227.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.227.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/HTTP_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.227.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/HTTP_127.14.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:17:16 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219836, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:17:16.961755 15246 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:17:16.985205 15246 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:17:17.830986 15246 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:17:17.831154 15246 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms
I20260924 03:17:17.831300 15246 mini_ranger_kms.cc:227] Using postgres at 127.14.227.148:57773
2026-09-24 03:17:18,250 [I] DB FLAVOR :POSTGRES
2026-09-24 03:17:18,250 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:17:18,250 [I] Checking connection
2026-09-24 03:17:18,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:17:18,503 [I] connection success
2026-09-24 03:17:18,503 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:17:18,503 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:17:18,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:17:18,740 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:17:18,740 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:17:18,740 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:17:18,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:57773/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:17:18,981 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:17:19.426159 15246 mini_ranger_kms.cc:310] Created kms service
I20260924 03:17:19.597551 15246 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:17:19.644131 15246 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:17:19 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219836, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:17:20.027488 15246 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:17:20.027803 15246 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms:/tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:17:20.027880 15246 mini_ranger_kms.cc:240] Using host: 127.14.227.148
I20260924 03:17:20.044576 15246 mini_ranger_kms.cc:276] Ranger KMS PID: 26468
I20260924 03:17:20.044795 15246 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.227.148:36391
Sep 24, 2026 3:17:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:17:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:17:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:17:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:17:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:17:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 03:17:22.353915 15246 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.392s user 0.113s sys 0.326s
I20260924 03:17:22.354246 15246 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:17:22.354480 15246 mini_ranger_kms.cc:401] 127.14.227.148:36391/kms/v1/keys
Sep 24 03:17:22 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219842, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.227.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 03:17:22 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219842, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.227.148@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
Sep 24 03:17:23 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219842, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.227.148@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
Sep 24 03:17:25 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:17:25.421547 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44059
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:44059
--ranger_config_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:36391/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:17:25.762571 26519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:17:25.762861 26519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:17:25.762955 26519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:17:25.770556 26519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:17:25.770671 26519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:17:25.770720 26519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:17:25.770782 26519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:17:25.770834 26519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:17:25.780761 26519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.227.148:36391/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:44059
--ranger_config_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:17:25.782066 26519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:17:25.783494 26519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:17:25.796041 26527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:17:25.796008 26524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:17:25.796325 26525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:17:25.796681 26519 server_base.cc:1034] running on GCE node
I20260924 03:17:25.797461 26519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:17:25.798908 26519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:17:25.800151 26519 hybrid_clock.cc:648] HybridClock initialized: now 1790219845800102 us; error 51 us; skew 500 ppm
I20260924 03:17:25.800486 26519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:17:25 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:17:25.807636 26519 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:17:25.809052 26519 webserver.cc:460] Webserver started at http://127.14.227.190:44933/ using document root <none> and password file <none>
I20260924 03:17:25.809729 26519 fs_manager.cc:359] Metadata directory not provided
I20260924 03:17:25.809871 26519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:17:25.810235 26519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:17:25 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219845, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:17:25.876024 26519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "3521560b15f64b2d8269f3f6a637355e"
format_stamp: "Formatted at 2026-09-24 03:17:25 on dist-test-slave-vd0p"
server_key: "6adf0dc96521aa8ce70a0550697373de"
server_key_iv: "118355a5f5596720085c72eeba6490f9"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:25.877170 26519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "3521560b15f64b2d8269f3f6a637355e"
format_stamp: "Formatted at 2026-09-24 03:17:25 on dist-test-slave-vd0p"
server_key: "6adf0dc96521aa8ce70a0550697373de"
server_key_iv: "118355a5f5596720085c72eeba6490f9"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:25.883997 26519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:17:25.907557 26534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:25.909363 26519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:17:25.909556 26519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "3521560b15f64b2d8269f3f6a637355e"
format_stamp: "Formatted at 2026-09-24 03:17:25 on dist-test-slave-vd0p"
server_key: "6adf0dc96521aa8ce70a0550697373de"
server_key_iv: "118355a5f5596720085c72eeba6490f9"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:25.909823 26519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:17:25.937242 26519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:17:25.942282 26519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:17:25.942538 26519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:17:25.959805 26519 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:44059
I20260924 03:17:25.959827 26585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:44059 every 8 connection(s)
I20260924 03:17:25.961154 26519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:17:25.964939 26586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:17:25.967526 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 26519
I20260924 03:17:25.967730 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:17:25.968371 15246 external_mini_cluster.cc:1428] Setting key 40f527e34f0b80a6cd202f7a435959f4
I20260924 03:17:25.977712 26586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e: Bootstrap starting.
I20260924 03:17:25.981097 26586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e: Neither blocks nor log segments found. Creating new log.
I20260924 03:17:25.982074 26586 log.cc:824] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:17:25 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:17:25.985245 26586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e: No bootstrap required, opened a new log
I20260924 03:17:25.992338 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:25.970432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.14.227.190:44059)
0924 03:17:25.971193 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:25.971208 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:25.971255 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:17:25.973461 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:25.973495 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:25.974037 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:25.974477 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:17:25.975533 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:25.977403 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:25.978707 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:25.979799 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:25.979969 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:25.983686 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:17:25.983696 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:17:25.983714 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:17:25.983755 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:17:25.987965 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:25.988611 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:25.988621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:25.988634 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:25.988816 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:25.989202 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:25.989206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:25.989210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:25.989905 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:17:25.990223 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:25.990570 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 03:17:25.990977 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":321,"threads_started":1}
I20260924 03:17:25.995983 26586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521560b15f64b2d8269f3f6a637355e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44059 } }
I20260924 03:17:25.996325 26586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:17:25.996403 26586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3521560b15f64b2d8269f3f6a637355e, State: Initialized, Role: FOLLOWER
I20260924 03:17:25.996908 26586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521560b15f64b2d8269f3f6a637355e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44059 } }
I20260924 03:17:25.997133 26586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:17:25.997238 26586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:17:25.997347 26586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:17:25.999424 26586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521560b15f64b2d8269f3f6a637355e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44059 } }
I20260924 03:17:25.999831 26586 leader_election.cc:302] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3521560b15f64b2d8269f3f6a637355e; no voters:
I20260924 03:17:26.000394 26586 leader_election.cc:288] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:17:26.000573 26591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:17:26.001711 26591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [term 1 LEADER]: Becoming Leader. State: Replica: 3521560b15f64b2d8269f3f6a637355e, State: Running, Role: LEADER
I20260924 03:17:26.002820 26591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521560b15f64b2d8269f3f6a637355e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44059 } }
I20260924 03:17:26.002965 26586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:17:26.005837 26593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3521560b15f64b2d8269f3f6a637355e. Latest consensus state: current_term: 1 leader_uuid: "3521560b15f64b2d8269f3f6a637355e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521560b15f64b2d8269f3f6a637355e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44059 } } }
I20260924 03:17:26.006235 26593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [sys.catalog]: This master's current role is: LEADER
I20260924 03:17:26.005960 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3521560b15f64b2d8269f3f6a637355e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3521560b15f64b2d8269f3f6a637355e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 44059 } } }
I20260924 03:17:26.006358 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e [sys.catalog]: This master's current role is: LEADER
I20260924 03:17:26.019208 26599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:17:26.019565 26586 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:17:26.030413 26599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:17:26.042032 26599 catalog_manager.cc:1347] Generated new cluster ID: 823901ce9e094950ad4aba49bd0c04d6
I20260924 03:17:26.042167 26599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:17:26.051743 26599 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:17:26.053467 26599 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:17:26.059406 26599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e: Generated new TSK 0
I20260924 03:17:26.060184 26599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:17:26.797922 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:36391/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44059
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:17:27 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:17:27.222191 26631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:17:27.222508 26631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:17:27.222700 26631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:17:27.230768 26631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:17:27.230919 26631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:17:27.231107 26631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:17:27.241672 26631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.227.148:36391/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:17:27.243120 26631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:17:27.244994 26631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:17:27.265905 26641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:17:27.272662 26644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:17:27.273518 26642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:17:27.274343 26631 server_base.cc:1034] running on GCE node
I20260924 03:17:27.275009 26631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:17:27.286224 26631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:17:27.288079 26631 hybrid_clock.cc:648] HybridClock initialized: now 1790219847287969 us; error 115 us; skew 500 ppm
I20260924 03:17:27.288513 26631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:17:27 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:17:27.292057 26631 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:17:27.295820 26631 webserver.cc:460] Webserver started at http://127.14.227.129:40449/ using document root <none> and password file <none>
I20260924 03:17:27.296536 26631 fs_manager.cc:359] Metadata directory not provided
I20260924 03:17:27.296692 26631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:17:27.297011 26631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:17:27 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:17:27.319218 26631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "62341a4d35d74bb7842869aaa321bc50"
format_stamp: "Formatted at 2026-09-24 03:17:27 on dist-test-slave-vd0p"
server_key: "51628658dd30e61e0eac416f008710e9"
server_key_iv: "96493a3cbb3ba453bf711f44e8600b5b"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:27.322175 26631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "62341a4d35d74bb7842869aaa321bc50"
format_stamp: "Formatted at 2026-09-24 03:17:27 on dist-test-slave-vd0p"
server_key: "51628658dd30e61e0eac416f008710e9"
server_key_iv: "96493a3cbb3ba453bf711f44e8600b5b"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:27.346366 26631 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.012s sys 0.005s
I20260924 03:17:27.379482 26651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:27.381539 26631 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.007s
I20260924 03:17:27.381791 26631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "62341a4d35d74bb7842869aaa321bc50"
format_stamp: "Formatted at 2026-09-24 03:17:27 on dist-test-slave-vd0p"
server_key: "51628658dd30e61e0eac416f008710e9"
server_key_iv: "96493a3cbb3ba453bf711f44e8600b5b"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:27.382148 26631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:17:27.435288 26631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:17:27.440325 26631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:17:27.440676 26631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:17:27.443356 26631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:17:27.445051 26631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:17:27.445147 26631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:27.445304 26631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:17:27.445379 26631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:27.522337 26631 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:45621
I20260924 03:17:27.522392 26763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:45621 every 8 connection(s)
I20260924 03:17:27.524003 26631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:17:27.528602 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 26631
I20260924 03:17:27.528811 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:17:27.529659 15246 external_mini_cluster.cc:1428] Setting key 7b48ac72f71acc3424866b452aad3ac3
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
Sep 24 03:17:27 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:17:27.598954 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:27.536439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44059 (local address 127.14.227.129:36507)
0924 03:17:27.561348 (+ 24909us) negotiation.cc:107] Waiting for socket to connect
0924 03:17:27.561413 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:17:27.563965 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:17:27.566278 (+ 2313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:17:27.566291 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:17:27.567606 (+ 1315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:17:27.568416 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:27.568438 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:27.570236 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:27.570241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:27.572319 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:27.572333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:27.573547 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:27.573554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:27.573828 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:27.574598 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:17:27.574630 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:17:27.587040 (+ 12410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:17:27.594187 (+ 7147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:17:27.594198 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:17:27.594218 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:17:27.594816 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:17:27.595318 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:17:27.595323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:17:27.595327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:17:27.595576 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:17:27.596315 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:17:27.596324 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:17:27.596766 (+ 442us) client_negotiation.cc:769] Sending connection context
0924 03:17:27.597055 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 03:17:27.597592 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24431,"thread_start_us":28519,"threads_started":1}
I20260924 03:17:27.600646 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:27.537021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:36507 (local address 127.14.227.190:44059)
0924 03:17:27.560496 (+ 23475us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:27.560503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:27.564442 (+ 3939us) server_negotiation.cc:407] Connection header received
0924 03:17:27.564511 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:27.564516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:27.564618 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:27.564738 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:17:27.568619 (+ 3881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:27.570028 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:27.572514 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:27.573125 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:27.573226 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:27.587394 (+ 14168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:17:27.587403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:17:27.587408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:17:27.587477 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:17:27.593983 (+ 6506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:27.595008 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:27.595014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:27.595018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:27.595174 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:27.595702 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:27.595768 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:27.595772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:27.596104 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:17:27.596245 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:27.599487 (+ 3242us) server_negotiation.cc:299] Negotiation successful
0924 03:17:27.599711 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23335,"thread_start_us":23509,"threads_started":1}
I20260924 03:17:27.602253 26764 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44059
I20260924 03:17:27.603003 26764 heartbeater.cc:464] Registering TS with master...
I20260924 03:17:27.606097 26764 heartbeater.cc:511] Master 127.14.227.190:44059 requested a full tablet report, sending...
I20260924 03:17:27.609534 26551 ts_manager.cc:194] Registered new tserver with Master: 62341a4d35d74bb7842869aaa321bc50 (127.14.227.129:45621)
I20260924 03:17:27.612424 26551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:36507
Sep 24 03:17:27 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:17:27.631421 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:36391/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44059
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:17:27 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219847, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
W20260924 03:17:28.117683 26770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:17:28.118085 26770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:17:28.118265 26770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:17:28.129652 26770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:17:28.129802 26770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:17:28.130034 26770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:17:28.146461 26770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.227.148:36391/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:17:28.148087 26770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:17:28.149894 26770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:17:28.165272 26784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:17:28.165463 26785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:17:28.165992 26787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:17:28.166247 26770 server_base.cc:1034] running on GCE node
I20260924 03:17:28.166882 26770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:17:28.167801 26770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:17:28.169036 26770 hybrid_clock.cc:648] HybridClock initialized: now 1790219848168980 us; error 55 us; skew 500 ppm
I20260924 03:17:28.169348 26770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:17:28 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219848, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:17:28.172570 26770 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:17:28.174122 26770 webserver.cc:460] Webserver started at http://127.14.227.130:42287/ using document root <none> and password file <none>
I20260924 03:17:28.174824 26770 fs_manager.cc:359] Metadata directory not provided
I20260924 03:17:28.174921 26770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:17:28.175244 26770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:17:28 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219848, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:17:28.192533 26770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "df2bd2680b494f2b955d24e89f778fbe"
format_stamp: "Formatted at 2026-09-24 03:17:28 on dist-test-slave-vd0p"
server_key: "1c25b0d5c46dafe601c9d807cabad9fc"
server_key_iv: "70f328c8dbf33b01182e5d9879db757b"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:28.193658 26770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "df2bd2680b494f2b955d24e89f778fbe"
format_stamp: "Formatted at 2026-09-24 03:17:28 on dist-test-slave-vd0p"
server_key: "1c25b0d5c46dafe601c9d807cabad9fc"
server_key_iv: "70f328c8dbf33b01182e5d9879db757b"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:28.199659 26770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:17:28.216243 26794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:28.218127 26770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:17:28.218325 26770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "df2bd2680b494f2b955d24e89f778fbe"
format_stamp: "Formatted at 2026-09-24 03:17:28 on dist-test-slave-vd0p"
server_key: "1c25b0d5c46dafe601c9d807cabad9fc"
server_key_iv: "70f328c8dbf33b01182e5d9879db757b"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:28.218624 26770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:17:28.246634 26770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:17:28.251209 26770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:17:28.251516 26770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:17:28.252558 26770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:17:28.253906 26770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:17:28.254036 26770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:28.254182 26770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:17:28.254256 26770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:28.293476 26770 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:42623
I20260924 03:17:28.293498 26906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:42623 every 8 connection(s)
I20260924 03:17:28.294814 26770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:17:28.301030 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 26770
I20260924 03:17:28.301265 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:17:28.301688 15246 external_mini_cluster.cc:1428] Setting key 360f9affee4785cc2be3f22de090f3d6
Sep 24 03:17:28 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219848, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:17:28.318681 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.298034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44059 (local address 127.14.227.130:51703)
0924 03:17:28.299215 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0924 03:17:28.299281 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 03:17:28.301571 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:17:28.302480 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:17:28.302493 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:17:28.303087 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:17:28.303939 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.303958 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.305112 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.305118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.306472 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.306483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.307340 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.307351 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.307565 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.308342 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:17:28.308369 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:17:28.310495 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:17:28.313908 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:17:28.313920 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:17:28.313940 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:17:28.314570 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:17:28.315031 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:17:28.315039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:17:28.315045 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:17:28.315290 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:17:28.315912 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:17:28.315922 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:17:28.316338 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 03:17:28.316633 (+ 295us) client_negotiation.cc:240] Negotiation successful
0924 03:17:28.317189 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":701,"threads_started":1}
I20260924 03:17:28.318766 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.297402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:51703 (local address 127.14.227.190:44059)
0924 03:17:28.297842 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:28.297848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:28.299424 (+ 1576us) server_negotiation.cc:407] Connection header received
0924 03:17:28.301992 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:28.301998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:28.302090 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:28.302204 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:17:28.304149 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.304970 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.306611 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.307006 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.307132 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.310709 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:17:28.310716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:17:28.310721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:17:28.310761 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:17:28.313772 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:28.314714 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:28.314742 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:28.314746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:28.314917 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:28.315422 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:28.315427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:28.315431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:28.315775 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:17:28.315923 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:28.318156 (+ 2233us) server_negotiation.cc:299] Negotiation successful
0924 03:17:28.318390 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":208,"threads_started":1}
I20260924 03:17:28.320830 26907 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44059
I20260924 03:17:28.321153 26907 heartbeater.cc:464] Registering TS with master...
I20260924 03:17:28.322008 26907 heartbeater.cc:511] Master 127.14.227.190:44059 requested a full tablet report, sending...
I20260924 03:17:28.323665 26551 ts_manager.cc:194] Registered new tserver with Master: df2bd2680b494f2b955d24e89f778fbe (127.14.227.130:42623)
I20260924 03:17:28.324782 26551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:51703
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:17:28.356515 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:36391/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44059
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:17:28.616612 26764 heartbeater.cc:503] Master 127.14.227.190:44059 was elected leader, sending a full tablet report...
W20260924 03:17:28.689116 26913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:17:28.689384 26913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:17:28.689476 26913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:17:28.696616 26913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:17:28.696719 26913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:17:28.696861 26913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:17:28.706367 26913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.227.148:36391/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:17:28.707576 26913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:17:28.708853 26913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:17:28.722321 26918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:17:28.722267 26919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:17:28.722685 26921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:17:28.722962 26913 server_base.cc:1034] running on GCE node
I20260924 03:17:28.723628 26913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:17:28.724488 26913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:17:28.725726 26913 hybrid_clock.cc:648] HybridClock initialized: now 1790219848725650 us; error 66 us; skew 500 ppm
I20260924 03:17:28.726128 26913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:17:28 dist-test-slave-vd0p krb5kdc[26075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219848, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:17:28.730384 26913 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:17:28.732429 26913 webserver.cc:460] Webserver started at http://127.14.227.131:38451/ using document root <none> and password file <none>
I20260924 03:17:28.733398 26913 fs_manager.cc:359] Metadata directory not provided
I20260924 03:17:28.733548 26913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:17:28.733937 26913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:17:28 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219848, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:17:28.749641 26913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "706f8475cb2041bd8df5acf9dd8d26f3"
format_stamp: "Formatted at 2026-09-24 03:17:28 on dist-test-slave-vd0p"
server_key: "ee0d16164082730314d91ce9bf48bdf6"
server_key_iv: "dd43cc32fa750d96f0abd230a453b109"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:28.750751 26913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "706f8475cb2041bd8df5acf9dd8d26f3"
format_stamp: "Formatted at 2026-09-24 03:17:28 on dist-test-slave-vd0p"
server_key: "ee0d16164082730314d91ce9bf48bdf6"
server_key_iv: "dd43cc32fa750d96f0abd230a453b109"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:28.756907 26913 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 03:17:28.770723 26928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:28.772120 26913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:17:28.772310 26913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "706f8475cb2041bd8df5acf9dd8d26f3"
format_stamp: "Formatted at 2026-09-24 03:17:28 on dist-test-slave-vd0p"
server_key: "ee0d16164082730314d91ce9bf48bdf6"
server_key_iv: "dd43cc32fa750d96f0abd230a453b109"
server_key_version: "kuduclusterkey@0"
I20260924 03:17:28.772514 26913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:17:28.791575 26913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:17:28.795214 26913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:17:28.795493 26913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:17:28.796353 26913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:17:28.797596 26913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:17:28.797696 26913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:28.797843 26913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:17:28.797911 26913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:17:28.831871 26913 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:38907
I20260924 03:17:28.831899 27040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:38907 every 8 connection(s)
I20260924 03:17:28.833137 26913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:17:28.837740 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 26913
I20260924 03:17:28.837925 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:17:28.838541 15246 external_mini_cluster.cc:1428] Setting key c4273c3c6aa859293ef336c3956297dc
Sep 24 03:17:28 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219848, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:17:28.851462 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.835485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:58495 (local address 127.14.227.190:44059)
0924 03:17:28.835904 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:28.835910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:28.836919 (+ 1009us) server_negotiation.cc:407] Connection header received
0924 03:17:28.838840 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:28.838846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:28.838932 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:28.839059 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:17:28.840556 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.841247 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.842722 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.843524 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.843670 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.846663 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:17:28.846667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:17:28.846671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:17:28.846705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:17:28.849043 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:28.849653 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:28.849657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:28.849659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:28.849763 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:28.850167 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:28.850170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:28.850172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:28.850422 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:17:28.850523 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:28.851082 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 03:17:28.851218 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":199,"threads_started":1}
I20260924 03:17:28.852591 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.836121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:44059 (local address 127.14.227.131:58495)
0924 03:17:28.836761 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 03:17:28.836803 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:17:28.838543 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:17:28.839254 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:17:28.839262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:17:28.839694 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:17:28.840393 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.840413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.841409 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.841413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.842564 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.842575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.843732 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.843739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.843970 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.844512 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:17:28.844531 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:17:28.846483 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:17:28.849171 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:17:28.849178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:17:28.849187 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:17:28.849554 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:17:28.849847 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:17:28.849850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:17:28.849852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:17:28.850074 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:17:28.850517 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:17:28.850522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:17:28.850784 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 03:17:28.850997 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 03:17:28.851329 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":270,"threads_started":1}
I20260924 03:17:28.853767 27041 heartbeater.cc:347] Connected to a master server at 127.14.227.190:44059
I20260924 03:17:28.854065 27041 heartbeater.cc:464] Registering TS with master...
I20260924 03:17:28.854720 27041 heartbeater.cc:511] Master 127.14.227.190:44059 requested a full tablet report, sending...
I20260924 03:17:28.855984 26551 ts_manager.cc:194] Registered new tserver with Master: 706f8475cb2041bd8df5acf9dd8d26f3 (127.14.227.131:38907)
I20260924 03:17:28.856736 26551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:58495
I20260924 03:17:28.865420 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:17:28.881242 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.869749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.14.227.190:44059)
0924 03:17:28.869890 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:28.869894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:28.870296 (+ 402us) server_negotiation.cc:407] Connection header received
0924 03:17:28.870424 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:28.870427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:28.870486 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:28.870558 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:17:28.872131 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.872744 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.874094 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.874602 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.874691 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.876915 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:17:28.876920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:17:28.876923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:17:28.876948 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:17:28.879363 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:28.879903 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:28.879906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:28.879908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:28.880013 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:28.880292 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:28.880295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:28.880297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:28.880507 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:17:28.880684 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:28.880910 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 03:17:28.881045 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 03:17:28.884750 26551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:17:28.891763 26551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:17:28.915016 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.906535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38907 (local address 127.0.0.1:39736)
0924 03:17:28.908036 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0924 03:17:28.908052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:17:28.908164 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:17:28.908779 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:17:28.908784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:17:28.908807 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:17:28.909105 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.909115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.910658 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.910665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.912516 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.912529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.913892 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.913898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.914195 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.914246 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 03:17:28.914420 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 03:17:28.914622 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1394,"mutex_wait_us":71,"thread_start_us":1228,"threads_started":1}
I20260924 03:17:28.915104 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.905584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.14.227.130:42623)
0924 03:17:28.906147 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:28.906155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:28.906178 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:17:28.906263 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:28.906267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:28.906407 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:28.906954 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:17:28.908050 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.909475 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.912515 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.913595 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.913799 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.914033 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:28.914579 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 03:17:28.914746 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260924 03:17:28.915977 27056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.905390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:42623 (local address 127.0.0.1:40952)
0924 03:17:28.905981 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 03:17:28.905999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:17:28.906124 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:17:28.906625 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:17:28.906632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:17:28.906669 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:17:28.907869 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.907882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.909661 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.909667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.912352 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.912365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.913836 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.913842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.914168 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.914245 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 03:17:28.914419 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 03:17:28.915535 (+ 1116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":1159,"spinlock_wait_cycles":384,"thread_start_us":311,"threads_started":1}
I20260924 03:17:28.916448 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.903818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45621 (local address 127.0.0.1:51380)
0924 03:17:28.905057 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0924 03:17:28.905093 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:17:28.905278 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:17:28.906625 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:17:28.906632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:17:28.906675 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:17:28.907761 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.907783 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.909683 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.909688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.912351 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.912365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.915784 (+ 3419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.915788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.915907 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.915929 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:17:28.916005 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 03:17:28.916143 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":149,"spinlock_wait_cycles":3712,"thread_start_us":923,"threads_started":1}
I20260924 03:17:28.917739 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.904073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.14.227.129:45621)
0924 03:17:28.904749 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:28.904761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:28.905200 (+ 439us) server_negotiation.cc:407] Connection header received
0924 03:17:28.906158 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:28.906167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:28.906400 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:28.907235 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:17:28.908020 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.909498 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.914130 (+ 4632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.915572 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.915777 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.915942 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:28.916075 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 03:17:28.917386 (+ 1311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":331,"threads_started":1}
I20260924 03:17:28.918174 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.906299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.14.227.131:38907)
0924 03:17:28.906783 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:28.906790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:28.908279 (+ 1489us) server_negotiation.cc:407] Connection header received
0924 03:17:28.908371 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:28.908380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:28.908603 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:28.908775 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:17:28.909293 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.910517 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.912660 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.913620 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.913829 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.914036 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:28.917631 (+ 3595us) server_negotiation.cc:299] Negotiation successful
0924 03:17:28.917800 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"spinlock_wait_cycles":17280,"thread_start_us":336,"threads_started":1}
I20260924 03:17:28.919956 26842 tablet_service.cc:1467] Processing CreateTablet for tablet dc70e19ab9c44c15a0c8e344956ba2f9 (DEFAULT_TABLE table=test-table [id=1a2dfe4a246b4c3390cccd77455f3e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:17:28.921080 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc70e19ab9c44c15a0c8e344956ba2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:17:28.920913 26976 tablet_service.cc:1467] Processing CreateTablet for tablet dc70e19ab9c44c15a0c8e344956ba2f9 (DEFAULT_TABLE table=test-table [id=1a2dfe4a246b4c3390cccd77455f3e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:17:28.922027 26976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc70e19ab9c44c15a0c8e344956ba2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:17:28.925577 26699 tablet_service.cc:1467] Processing CreateTablet for tablet dc70e19ab9c44c15a0c8e344956ba2f9 (DEFAULT_TABLE table=test-table [id=1a2dfe4a246b4c3390cccd77455f3e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:17:28.926888 26699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc70e19ab9c44c15a0c8e344956ba2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:17:28.931648 27060 tablet_bootstrap.cc:492] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Bootstrap starting.
I20260924 03:17:28.934748 27060 tablet_bootstrap.cc:654] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Neither blocks nor log segments found. Creating new log.
I20260924 03:17:28.936276 27060 log.cc:824] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Log is configured to *not* fsync() on all Append() calls
I20260924 03:17:28.936363 27061 tablet_bootstrap.cc:492] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: Bootstrap starting.
I20260924 03:17:28.940052 27063 tablet_bootstrap.cc:492] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Bootstrap starting.
I20260924 03:17:28.941278 27061 tablet_bootstrap.cc:654] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: Neither blocks nor log segments found. Creating new log.
I20260924 03:17:28.942489 27061 log.cc:824] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:17:28.943583 27063 tablet_bootstrap.cc:654] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Neither blocks nor log segments found. Creating new log.
I20260924 03:17:28.944803 27061 tablet_bootstrap.cc:492] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: No bootstrap required, opened a new log
I20260924 03:17:28.944762 27063 log.cc:824] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Log is configured to *not* fsync() on all Append() calls
I20260924 03:17:28.945000 27060 tablet_bootstrap.cc:492] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: No bootstrap required, opened a new log
I20260924 03:17:28.945086 27061 ts_tablet_manager.cc:1397] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260924 03:17:28.945252 27060 ts_tablet_manager.cc:1397] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260924 03:17:28.947525 27063 tablet_bootstrap.cc:492] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: No bootstrap required, opened a new log
I20260924 03:17:28.947799 27063 ts_tablet_manager.cc:1397] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 03:17:28.952850 27061 raft_consensus.cc:348] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:28.953171 27061 raft_consensus.cc:374] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:17:28.953290 27061 raft_consensus.cc:729] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706f8475cb2041bd8df5acf9dd8d26f3, State: Initialized, Role: FOLLOWER
I20260924 03:17:28.953819 27061 consensus_queue.cc:260] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:28.954799 27041 heartbeater.cc:503] Master 127.14.227.190:44059 was elected leader, sending a full tablet report...
I20260924 03:17:28.955500 27060 raft_consensus.cc:348] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:28.955963 27060 raft_consensus.cc:374] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:17:28.956104 27060 raft_consensus.cc:729] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2bd2680b494f2b955d24e89f778fbe, State: Initialized, Role: FOLLOWER
I20260924 03:17:28.957209 27061 ts_tablet_manager.cc:1428] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 03:17:28.957054 27060 consensus_queue.cc:260] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:28.958029 26907 heartbeater.cc:503] Master 127.14.227.190:44059 was elected leader, sending a full tablet report...
I20260924 03:17:28.958411 27063 raft_consensus.cc:348] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:28.958802 27063 raft_consensus.cc:374] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:17:28.958910 27063 raft_consensus.cc:729] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62341a4d35d74bb7842869aaa321bc50, State: Initialized, Role: FOLLOWER
I20260924 03:17:28.959555 27063 consensus_queue.cc:260] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:28.959930 27060 ts_tablet_manager.cc:1428] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260924 03:17:28.967193 27063 ts_tablet_manager.cc:1428] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260924 03:17:28.980834 27068 raft_consensus.cc:482] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:17:28.981079 27068 raft_consensus.cc:504] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:28.982692 27068 leader_election.cc:288] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2bd2680b494f2b955d24e89f778fbe (127.14.227.130:42623), 706f8475cb2041bd8df5acf9dd8d26f3 (127.14.227.131:38907)
I20260924 03:17:28.989355 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.983419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:59409 (local address 127.14.227.130:42623)
0924 03:17:28.983591 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:28.983595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:28.983861 (+ 266us) server_negotiation.cc:407] Connection header received
0924 03:17:28.984161 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:28.984166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:28.984236 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:28.984301 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:17:28.985010 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.985704 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.987553 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.988344 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.988513 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.988583 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:28.988987 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 03:17:28.989104 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:17:28.989370 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.984005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:58965 (local address 127.14.227.131:38907)
0924 03:17:28.984195 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:28.984202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:28.984435 (+ 233us) server_negotiation.cc:407] Connection header received
0924 03:17:28.984635 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:28.984640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:28.984706 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:28.984797 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:17:28.985300 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.986044 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.987640 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.988398 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.988558 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.988626 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:28.989017 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 03:17:28.989130 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:17:28.989537 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.982877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:42623 (local address 127.14.227.129:59409)
0924 03:17:28.983754 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 03:17:28.983772 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:17:28.983976 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:17:28.984452 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:17:28.984455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:17:28.984524 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:17:28.984799 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.984806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.985844 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.985850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.987380 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.987392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.988555 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.988561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.988773 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.988864 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 03:17:28.988964 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:17:28.989089 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":50,"spinlock_wait_cycles":27008,"thread_start_us":597,"threads_started":1}
I20260924 03:17:28.989604 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:28.983829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38907 (local address 127.14.227.129:58965)
0924 03:17:28.984343 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 03:17:28.984356 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:17:28.984497 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:17:28.984812 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:17:28.984816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:17:28.984839 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:17:28.985162 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.985170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.986181 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.986193 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.987493 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:17:28.987503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:17:28.988604 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:17:28.988610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:17:28.988821 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:28.988896 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 03:17:28.988996 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:17:28.989114 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":40,"thread_start_us":261,"threads_started":1}
I20260924 03:17:28.990490 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc70e19ab9c44c15a0c8e344956ba2f9" candidate_uuid: "62341a4d35d74bb7842869aaa321bc50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2bd2680b494f2b955d24e89f778fbe" is_pre_election: true
I20260924 03:17:28.990538 26996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc70e19ab9c44c15a0c8e344956ba2f9" candidate_uuid: "62341a4d35d74bb7842869aaa321bc50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" is_pre_election: true
I20260924 03:17:28.990940 26862 raft_consensus.cc:2454] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62341a4d35d74bb7842869aaa321bc50 in term 0.
I20260924 03:17:28.991022 26996 raft_consensus.cc:2454] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62341a4d35d74bb7842869aaa321bc50 in term 0.
I20260924 03:17:28.991629 26655 leader_election.cc:302] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62341a4d35d74bb7842869aaa321bc50, df2bd2680b494f2b955d24e89f778fbe; no voters:
I20260924 03:17:28.991937 27068 raft_consensus.cc:2781] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:17:28.992071 27068 raft_consensus.cc:482] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:17:28.992183 27068 raft_consensus.cc:3037] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:17:28.994750 27068 raft_consensus.cc:504] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:28.995453 27068 leader_election.cc:288] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [CANDIDATE]: Term 1 election: Requested vote from peers df2bd2680b494f2b955d24e89f778fbe (127.14.227.130:42623), 706f8475cb2041bd8df5acf9dd8d26f3 (127.14.227.131:38907)
I20260924 03:17:28.995836 26862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc70e19ab9c44c15a0c8e344956ba2f9" candidate_uuid: "62341a4d35d74bb7842869aaa321bc50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2bd2680b494f2b955d24e89f778fbe"
I20260924 03:17:28.995965 26996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc70e19ab9c44c15a0c8e344956ba2f9" candidate_uuid: "62341a4d35d74bb7842869aaa321bc50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706f8475cb2041bd8df5acf9dd8d26f3"
I20260924 03:17:28.996042 26862 raft_consensus.cc:3037] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:17:28.996150 26996 raft_consensus.cc:3037] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:17:28.999223 26996 raft_consensus.cc:2454] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62341a4d35d74bb7842869aaa321bc50 in term 1.
I20260924 03:17:28.999224 26862 raft_consensus.cc:2454] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62341a4d35d74bb7842869aaa321bc50 in term 1.
I20260924 03:17:28.999568 26655 leader_election.cc:302] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62341a4d35d74bb7842869aaa321bc50, df2bd2680b494f2b955d24e89f778fbe; no voters:
I20260924 03:17:28.999874 27068 raft_consensus.cc:2781] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:17:29.000290 27068 raft_consensus.cc:686] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 1 LEADER]: Becoming Leader. State: Replica: 62341a4d35d74bb7842869aaa321bc50, State: Running, Role: LEADER
I20260924 03:17:29.001091 27068 consensus_queue.cc:237] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } }
I20260924 03:17:29.005741 26550 catalog_manager.cc:5495] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62341a4d35d74bb7842869aaa321bc50 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "62341a4d35d74bb7842869aaa321bc50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62341a4d35d74bb7842869aaa321bc50" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 } health_report { overall_health: UNKNOWN } } }
I20260924 03:17:29.026897 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:29.021444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.14.227.129:45621)
0924 03:17:29.021663 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:29.021668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:29.021685 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:17:29.021768 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:29.021772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:29.021846 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:29.022068 (+ 222us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:17:29.022467 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:29.023210 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:29.024685 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:29.025227 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:29.025323 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:29.025752 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:17:29.026330 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:17:29.026434 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:29.026560 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 03:17:29.026622 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 03:17:29.034353 26765 tablet.cc:2369] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:17:29.034437 26996 raft_consensus.cc:1261] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 1 FOLLOWER]: Refusing update from remote peer 62341a4d35d74bb7842869aaa321bc50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:17:29.034639 26862 raft_consensus.cc:1261] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 1 FOLLOWER]: Refusing update from remote peer 62341a4d35d74bb7842869aaa321bc50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:17:29.035399 27071 consensus_queue.cc:1035] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "706f8475cb2041bd8df5acf9dd8d26f3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:17:29.035724 27068 consensus_queue.cc:1035] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2bd2680b494f2b955d24e89f778fbe" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:17:29.046897 26908 tablet.cc:2369] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:17:29.060808 26551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:17:29.061087 26551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:17:29.064544 26551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3521560b15f64b2d8269f3f6a637355e: Sending DeleteTablet for 3 replicas of tablet dc70e19ab9c44c15a0c8e344956ba2f9
I20260924 03:17:29.065702 26699 tablet_service.cc:1514] Processing DeleteTablet for tablet dc70e19ab9c44c15a0c8e344956ba2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:17:29 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:51380
I20260924 03:17:29.065711 26842 tablet_service.cc:1514] Processing DeleteTablet for tablet dc70e19ab9c44c15a0c8e344956ba2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:17:29 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:40952
I20260924 03:17:29.065828 26976 tablet_service.cc:1514] Processing DeleteTablet for tablet dc70e19ab9c44c15a0c8e344956ba2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:17:29 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:39736
I20260924 03:17:29.067884 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 26631
I20260924 03:17:29.078119 27085 tablet_replica.cc:330] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: stopping tablet replica
I20260924 03:17:29.078737 27085 raft_consensus.cc:2229] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:17:29.079785 27085 raft_consensus.cc:2258] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:17:29.082162 27085 ts_tablet_manager.cc:1905] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:17:29.082801 27084 tablet_replica.cc:330] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: stopping tablet replica
I20260924 03:17:29.083228 27084 raft_consensus.cc:2229] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:17:29.084183 27086 tablet_replica.cc:330] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: stopping tablet replica
I20260924 03:17:29.084365 27084 raft_consensus.cc:2258] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:17:29.085124 27086 raft_consensus.cc:2229] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:17:29.085632 27086 raft_consensus.cc:2258] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:17:29.087003 27084 ts_tablet_manager.cc:1905] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:17:29.088279 27086 ts_tablet_manager.cc:1905] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:17:29 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:17:29.093878 27084 ts_tablet_manager.cc:1918] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:17:29.094139 27084 log.cc:1196] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/dc70e19ab9c44c15a0c8e344956ba2f9
I20260924 03:17:29.094697 27084 ts_tablet_manager.cc:1939] T dc70e19ab9c44c15a0c8e344956ba2f9 P df2bd2680b494f2b955d24e89f778fbe: Deleting consensus metadata
I20260924 03:17:29.096603 26538 catalog_manager.cc:4841] TS df2bd2680b494f2b955d24e89f778fbe (127.14.227.130:42623): tablet dc70e19ab9c44c15a0c8e344956ba2f9 (table test-table [id=1a2dfe4a246b4c3390cccd77455f3e1a]) successfully deleted
I20260924 03:17:29.096946 27086 ts_tablet_manager.cc:1918] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:17:29.097136 27086 log.cc:1196] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/dc70e19ab9c44c15a0c8e344956ba2f9
I20260924 03:17:29.097666 27086 ts_tablet_manager.cc:1939] T dc70e19ab9c44c15a0c8e344956ba2f9 P 706f8475cb2041bd8df5acf9dd8d26f3: Deleting consensus metadata
I20260924 03:17:29.099299 26538 catalog_manager.cc:4841] TS 706f8475cb2041bd8df5acf9dd8d26f3 (127.14.227.131:38907): tablet dc70e19ab9c44c15a0c8e344956ba2f9 (table test-table [id=1a2dfe4a246b4c3390cccd77455f3e1a]) successfully deleted
I20260924 03:17:29.099522 27085 ts_tablet_manager.cc:1918] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:17:29.099735 27085 log.cc:1196] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/dc70e19ab9c44c15a0c8e344956ba2f9
I20260924 03:17:29.100215 27085 ts_tablet_manager.cc:1939] T dc70e19ab9c44c15a0c8e344956ba2f9 P 62341a4d35d74bb7842869aaa321bc50: Deleting consensus metadata
I20260924 03:17:29.101078 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:29.069290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.14.227.129:45621)
0924 03:17:29.069503 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:29.069512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:29.074529 (+ 5017us) server_negotiation.cc:407] Connection header received
0924 03:17:29.075355 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:29.075363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:29.075464 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:29.075551 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:17:29.079884 (+ 4333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:29.081113 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:29.086002 (+ 4889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:29.086859 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:29.087001 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:29.092425 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:17:29.092439 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:17:29.092458 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:17:29.092502 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:17:29.096795 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:29.097872 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:29.097881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:29.097884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:29.098099 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:29.099076 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:29.099081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:29.099085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:29.099534 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:17:29.099856 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:29.100472 (+ 616us) server_negotiation.cc:299] Negotiation successful
0924 03:17:29.100716 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:17:29.101761 26759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:17:29.102705 26538 catalog_manager.cc:4841] TS 62341a4d35d74bb7842869aaa321bc50 (127.14.227.129:45621): tablet dc70e19ab9c44c15a0c8e344956ba2f9 (table test-table [id=1a2dfe4a246b4c3390cccd77455f3e1a]) successfully deleted
I20260924 03:17:29.198530 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 26631
I20260924 03:17:29.216320 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 26770
Sep 24 03:17:29 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:17:29.231413 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:29.216992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.14.227.130:42623)
0924 03:17:29.217164 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:29.217171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:29.217264 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:17:29.217466 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:29.217471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:29.217552 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:29.217657 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:17:29.218916 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:29.219629 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:29.220880 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:29.221265 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:29.221406 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:29.225726 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:17:29.225740 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:17:29.225755 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:17:29.225787 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:17:29.228596 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:29.229238 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:29.229255 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:29.229258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:29.229374 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:29.229716 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:29.229723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:29.229726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:29.230196 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:17:29.230569 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:29.230875 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 03:17:29.231081 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:17:29.232016 26902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:17:29.317071 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 26770
I20260924 03:17:29.332837 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 26913
Sep 24 03:17:29 dist-test-slave-vd0p krb5kdc[26075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:17:29.347214 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:17:29.333585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.14.227.131:38907)
0924 03:17:29.333786 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 03:17:29.333791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:17:29.333810 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:17:29.334073 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:17:29.334077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:17:29.334153 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:17:29.334244 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:17:29.335566 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:29.336285 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:29.337497 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:17:29.337838 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:17:29.337923 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:17:29.342146 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:17:29.342160 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:17:29.342176 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:17:29.342205 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:17:29.344776 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:29.345379 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:29.345386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:29.345388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:29.345516 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:17:29.345858 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:17:29.345861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:17:29.345863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:17:29.346234 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:17:29.346440 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 03:17:29.346819 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 03:17:29.346964 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:17:29.347873 27036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:17:29.436185 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 26913
I20260924 03:17:29.451779 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 26519
I20260924 03:17:29.452832 26581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:17:29.560953 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 26519
I20260924 03:17:29.582294 15246 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:17:29.916853 15246 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:17:29.917090 15246 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:17:30.253527 15246 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:17:30.254 UTC [26105] LOG: received smart shutdown request
2026-09-24 03:17:30.258 UTC [26105] LOG: background worker "logical replication launcher" (PID 26114) exited with exit code 1
2026-09-24 03:17:30.258 UTC [26109] LOG: shutting down
2026-09-24 03:17:30.426 UTC [26105] LOG: database system is shut down
2026-09-24T03:17:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48352 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:17:30 dist-test-slave-vd0p krb5kdc[27100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:17:30 dist-test-slave-vd0p krb5kdc[27100](info): set up 2 sockets
Sep 24 03:17:30 dist-test-slave-vd0p krb5kdc[27100](info): commencing operation
krb5kdc: starting...
W20260924 03:17:31.642228 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:17:31 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:17:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/rangeradmin_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/rangeradmin_127.14.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/rangerlookup_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/rangerlookup_127.14.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/HTTP_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/HTTP_127.14.227.148@KRBTEST.COM.keytab.
I20260924 03:17:31.953023 15246 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1C3avF/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/postgres -l logfile start
2026-09-24 03:17:34.437 UTC [27130] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 03:17:34.437 UTC [27130] LOG: listening on IPv4 address "127.14.227.148", port 45501
2026-09-24 03:17:34.440 UTC [27130] LOG: listening on Unix socket "/tmp/.s.PGSQL.45501"
2026-09-24 03:17:34.460 UTC [27133] LOG: database system was shut down at 2026-09-24 03:17:33 UTC
2026-09-24 03:17:34.464 UTC [27130] LOG: database system is ready to accept connections
I20260924 03:17:35.426024 15246 mini_postgres.cc:96] Postgres bound to 45501
2026-09-24 03:17:35.447 UTC [27141] FATAL: database "slave" does not exist
127.14.227.148:45501 - accepting connections
I20260924 03:17:35.447767 15246 mini_ranger.cc:161] Starting Ranger...
I20260924 03:17:35.472615 15246 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 03:17:35.932269 15246 mini_ranger.cc:87] Created ranger Postgres database
I20260924 03:17:35.932387 15246 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-admin
2026-09-24 03:17:36,208 [I] DB FLAVOR :POSTGRES
2026-09-24 03:17:36,209 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:17:36,209 [I] Checking connection..
2026-09-24 03:17:36,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 03:17:36,440 [I] Checking connection passed.
2026-09-24 03:17:36,440 [I] --------- Verifying version history table ---------
2026-09-24 03:17:36,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:17:36,701 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:17:36,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:17:36,960 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 03:17:36,960 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 03:17:36,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 03:17:37.185 UTC [27205] WARNING: there is no transaction in progress
2026-09-24 03:17:37,191 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 03:17:37,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 03:17:37,416 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 03:17:37,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:17:37,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 03:17:37,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 03:17:37,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vd0p.c.gcp-upstream.internal','N') ;"
2026-09-24 03:17:38,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:17:38,366 [I] Table x_portal_user does not exist in database ranger
2026-09-24 03:17:38,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:17:38,607 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 03:17:38,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:17:38,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:17:39,114 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 03:17:39,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 03:17:40.070 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.086 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.101 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.116 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.132 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.152 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.208 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.215 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.229 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.239 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.249 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.262 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.271 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.277 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.283 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40.289 UTC [27313] WARNING: there is no transaction in progress
2026-09-24 03:17:40,926 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 03:17:40,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 03:17:41,177 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 03:17:41,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 03:17:41,424 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 03:17:41,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 03:17:41,660 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 03:17:41,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 03:17:41,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 03:17:42,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vd0p.c.gcp-upstream.internal';"
2026-09-24 03:17:42,365 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 03:17:42,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 03:17:42,609 [I] DB_PATCHES have already been applied
I20260924 03:17:42.615263 15246 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-admin:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1C3avF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 03:17:42.615396 15246 mini_ranger.cc:193] Using host: 127.14.227.148
I20260924 03:17:42.629703 15246 mini_ranger.cc:227] Ranger admin URL: http://127.14.227.148:47155
Sep 24, 2026 3:17:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:17:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 03:17:43.761446 15246 mini_ranger.cc:160] Time spent starting Ranger: real 8.314s user 0.000s sys 0.037s
Sep 24, 2026 3:18:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:18:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:18:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:18:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:18:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 03:18:05.523440 15246 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.227.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.227.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/rangerkms_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.227.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/rangerkms_127.14.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.227.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.227.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.227.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/HTTP_127.14.227.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.227.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/HTTP_127.14.227.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 03:18:05 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219885, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 03:18:05.722273 15246 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 03:18:05.742760 15246 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 03:18:06.631297 15246 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 03:18:06.631434 15246 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms
I20260924 03:18:06.631512 15246 mini_ranger_kms.cc:227] Using postgres at 127.14.227.148:45501
2026-09-24 03:18:06,990 [I] DB FLAVOR :POSTGRES
2026-09-24 03:18:06,990 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 03:18:06,990 [I] Checking connection
2026-09-24 03:18:06,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 03:18:07,243 [I] connection success
2026-09-24 03:18:07,243 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 03:18:07,243 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 03:18:07,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 03:18:07,472 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 03:18:07,472 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 03:18:07,472 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 03:18:07,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.227.148:45501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 03:18:07,731 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 03:18:08.204473 15246 mini_ranger_kms.cc:310] Created kms service
I20260924 03:18:08.415235 15246 mini_ranger_kms.cc:326] Created kudu user
I20260924 03:18:08.459262 15246 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 03:18:08 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219885, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:08.857234 15246 mini_ranger_kms.cc:384] Added ranger policy
I20260924 03:18:08.857544 15246 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms:/tmp/dist-test-task1C3avF/build/asan/bin/postgresql.jar:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1C3avF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 03:18:08.857596 15246 mini_ranger_kms.cc:240] Using host: 127.14.227.148
I20260924 03:18:08.874050 15246 mini_ranger_kms.cc:276] Ranger KMS PID: 27493
I20260924 03:18:08.874190 15246 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.227.148:52709
Sep 24, 2026 3:18:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:18:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:18:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:18:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:18:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:18:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 03:18:10 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.227.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:11.134383 15246 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.412s user 0.094s sys 0.278s
I20260924 03:18:11.134650 15246 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 03:18:11.134850 15246 mini_ranger_kms.cc:401] 127.14.227.148:52709/kms/v1/keys
Sep 24 03:18:11 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.227.148@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
Sep 24 03:18:11 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219890, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.227.148@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
Sep 24 03:18:13 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:18:14.009196 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33897
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:33897
--ranger_config_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:52709/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:14.334945 27544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:14.335300 27544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:14.335460 27544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:14.342653 27544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:14.342790 27544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:14.342890 27544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:14.342986 27544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:14.343062 27544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:14.352941 27544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.227.148:52709/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:33897
--ranger_config_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:14.354300 27544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:14.355571 27544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:14.368695 27550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:14.368829 27552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:14.369416 27544 server_base.cc:1034] running on GCE node
W20260924 03:18:14.369462 27549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:14.370579 27544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:14.372354 27544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:14.373641 27544 hybrid_clock.cc:648] HybridClock initialized: now 1790219894373581 us; error 61 us; skew 500 ppm
I20260924 03:18:14.374058 27544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:14 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:14.382139 27544 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:18:14.383814 27544 webserver.cc:460] Webserver started at http://127.14.227.190:36779/ using document root <none> and password file <none>
I20260924 03:18:14.384471 27544 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:14.384584 27544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:14.384919 27544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:18:14 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:14.433849 27544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "b6aec7a05b3147ea90adaa3d91df765e"
format_stamp: "Formatted at 2026-09-24 03:18:14 on dist-test-slave-vd0p"
server_key: "f89151b287673eac75e318ca515a4143"
server_key_iv: "038d5ef6fff469b21cbace60f8852a6b"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:14.434857 27544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "b6aec7a05b3147ea90adaa3d91df765e"
format_stamp: "Formatted at 2026-09-24 03:18:14 on dist-test-slave-vd0p"
server_key: "f89151b287673eac75e318ca515a4143"
server_key_iv: "038d5ef6fff469b21cbace60f8852a6b"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:14.440511 27544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 03:18:14.456979 27559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:14.458690 27544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:18:14.458878 27544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "b6aec7a05b3147ea90adaa3d91df765e"
format_stamp: "Formatted at 2026-09-24 03:18:14 on dist-test-slave-vd0p"
server_key: "f89151b287673eac75e318ca515a4143"
server_key_iv: "038d5ef6fff469b21cbace60f8852a6b"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:14.459141 27544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:14.485431 27544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:14.489459 27544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:14.489790 27544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:14.509719 27544 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:33897
I20260924 03:18:14.509721 27610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:33897 every 8 connection(s)
I20260924 03:18:14.511145 27544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:18:14.515152 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 27544
I20260924 03:18:14.515316 27611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:14.515404 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:18:14.516283 15246 external_mini_cluster.cc:1428] Setting key d2bb7b98ad4d14865fc932e07b706b69
I20260924 03:18:14.529681 27611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Bootstrap starting.
I20260924 03:18:14.532819 27611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Neither blocks nor log segments found. Creating new log.
Sep 24 03:18:14 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:14.533995 27611 log.cc:824] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:14.536912 27611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: No bootstrap required, opened a new log
I20260924 03:18:14.543148 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:14.518573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.14.227.190:33897)
0924 03:18:14.519438 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:14.519459 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:14.519513 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:18:14.522355 (+ 2842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:14.522383 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:14.522958 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:14.523403 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:14.524899 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:14.526937 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:14.528803 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:14.529808 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:14.530062 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:14.534602 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:14.534611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:14.534628 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:14.534671 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:14.538236 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:14.539078 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:14.539089 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:14.539097 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:14.539283 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:14.539721 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:14.539726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:14.539728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:14.540398 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:14.540746 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:14.541184 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 03:18:14.541625 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":458,"threads_started":1}
I20260924 03:18:14.545575 27611 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } }
I20260924 03:18:14.546017 27611 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:14.546128 27611 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6aec7a05b3147ea90adaa3d91df765e, State: Initialized, Role: FOLLOWER
I20260924 03:18:14.546737 27611 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } }
I20260924 03:18:14.546972 27611 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:18:14.547106 27611 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:18:14.547259 27611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:14.549363 27611 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } }
I20260924 03:18:14.549857 27611 leader_election.cc:302] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6aec7a05b3147ea90adaa3d91df765e; no voters:
I20260924 03:18:14.550596 27611 leader_election.cc:288] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:18:14.550704 27616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:14.553005 27611 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:14.557699 27616 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 1 LEADER]: Becoming Leader. State: Replica: b6aec7a05b3147ea90adaa3d91df765e, State: Running, Role: LEADER
I20260924 03:18:14.558461 27616 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } }
I20260924 03:18:14.561622 27617 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6aec7a05b3147ea90adaa3d91df765e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } } }
I20260924 03:18:14.561997 27617 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:14.562472 27618 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6aec7a05b3147ea90adaa3d91df765e. Latest consensus state: current_term: 1 leader_uuid: "b6aec7a05b3147ea90adaa3d91df765e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } } }
I20260924 03:18:14.562755 27618 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:14.568300 27623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:18:14.574529 27623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:18:14.575244 27611 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 03:18:14.593540 27623 catalog_manager.cc:1347] Generated new cluster ID: ef28dcc82f7347d5a7822fa273f6e297
I20260924 03:18:14.593686 27623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:18:14.627242 27623 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:18:14.629426 27623 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:18:14.640051 27623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Generated new TSK 0
I20260924 03:18:14.641180 27623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:18:15.450464 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:52709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33897
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:18:15 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:18:15.881546 27656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:15.881836 27656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:15.882030 27656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:15.889281 27656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:15.889415 27656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:15.889600 27656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:18:15.899530 27656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.227.148:52709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:15.901070 27656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:15.902560 27656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:15.924031 27667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:15.928563 27666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:15.928828 27656 server_base.cc:1034] running on GCE node
W20260924 03:18:15.929813 27669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:15.930671 27656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:15.932550 27656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:15.933830 27656 hybrid_clock.cc:648] HybridClock initialized: now 1790219895933734 us; error 84 us; skew 500 ppm
I20260924 03:18:15.934266 27656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:15 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:15.938351 27656 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:18:15.943383 27656 webserver.cc:460] Webserver started at http://127.14.227.129:40917/ using document root <none> and password file <none>
I20260924 03:18:15.944247 27656 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:15.944384 27656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:15.944738 27656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:18:15 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:15.970436 27656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "60df20d8272f4943a0598d06f8e5d960"
format_stamp: "Formatted at 2026-09-24 03:18:15 on dist-test-slave-vd0p"
server_key: "8c0fc2f7394a58771476e5b8b9dfc777"
server_key_iv: "1ce401021bc8e6149bac6463ef4519f9"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:15.971481 27656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "60df20d8272f4943a0598d06f8e5d960"
format_stamp: "Formatted at 2026-09-24 03:18:15 on dist-test-slave-vd0p"
server_key: "8c0fc2f7394a58771476e5b8b9dfc777"
server_key_iv: "1ce401021bc8e6149bac6463ef4519f9"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:15.978451 27656 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 03:18:16.010462 27676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:16.012462 27656 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.011s
I20260924 03:18:16.012715 27656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "60df20d8272f4943a0598d06f8e5d960"
format_stamp: "Formatted at 2026-09-24 03:18:15 on dist-test-slave-vd0p"
server_key: "8c0fc2f7394a58771476e5b8b9dfc777"
server_key_iv: "1ce401021bc8e6149bac6463ef4519f9"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:16.013046 27656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:16.054538 27656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:16.066751 27656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:16.067154 27656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:16.068611 27656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:16.070384 27656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:16.070511 27656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:16.070672 27656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:16.070744 27656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:16.142351 27656 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:36517
I20260924 03:18:16.142397 27788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:36517 every 8 connection(s)
I20260924 03:18:16.144089 27656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:18:16.154425 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 27656
I20260924 03:18:16.154634 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:18:16.155449 15246 external_mini_cluster.cc:1428] Setting key a625e8dd1360725d3e5ccf9293f5ed5d
Sep 24 03:18:16 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
I20260924 03:18:16.196996 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:16.150356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33897 (local address 127.14.227.129:59583)
0924 03:18:16.163826 (+ 13470us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:16.163904 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:16.167604 (+ 3700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:16.177686 (+ 10082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:16.177699 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:16.178405 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:16.179254 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:16.179278 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:16.180957 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:16.180963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:16.183021 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:16.183036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:16.184245 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:16.184252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:16.184519 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:16.185487 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:16.185515 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:16.188329 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:16.192227 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:16.192236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:16.192269 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:16.192886 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:16.193332 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:16.193336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:16.193340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:16.193569 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:16.194291 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:16.194299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:16.194737 (+ 438us) client_negotiation.cc:769] Sending connection context
0924 03:18:16.195027 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 03:18:16.195551 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13120,"thread_start_us":14882,"threads_started":1}
I20260924 03:18:16.198174 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:16.149735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:59583 (local address 127.14.227.190:33897)
0924 03:18:16.176409 (+ 26674us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:16.176416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:16.176439 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:18:16.176503 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:16.176509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:16.176599 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:16.179447 (+ 2848us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:16.179624 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:16.180740 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:16.183207 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:16.183841 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:16.183964 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:16.188605 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:16.188612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:16.188617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:16.188655 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:16.192090 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:16.193036 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:16.193041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:16.193045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:16.193200 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:16.193715 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:16.193720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:16.193723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:16.194083 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:16.194218 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:16.197498 (+ 3280us) server_negotiation.cc:299] Negotiation successful
0924 03:18:16.197756 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26532,"thread_start_us":26969,"threads_started":1}
I20260924 03:18:16.200697 27789 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33897
I20260924 03:18:16.201018 27789 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:16.201879 27789 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
I20260924 03:18:16.204414 27576 ts_manager.cc:194] Registered new tserver with Master: 60df20d8272f4943a0598d06f8e5d960 (127.14.227.129:36517)
I20260924 03:18:16.206578 27576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:59583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:18:16.225739 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:52709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33897
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:18:16 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
Sep 24 03:18:16 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219895, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
W20260924 03:18:16.671258 27795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:16.671787 27795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:16.672047 27795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:16.683486 27795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:16.683703 27795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:16.683945 27795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:18:16.699821 27795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.227.148:52709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:16.701869 27795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:16.703662 27795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:16.738693 27802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:16.742386 27803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:16.742568 27806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:16.744272 27795 server_base.cc:1034] running on GCE node
I20260924 03:18:16.745931 27795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:16.754243 27795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:16.765009 27795 hybrid_clock.cc:648] HybridClock initialized: now 1790219896764885 us; error 127 us; skew 500 ppm
I20260924 03:18:16.765404 27795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:16 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219896, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:16.770437 27795 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:18:16.791245 27795 webserver.cc:460] Webserver started at http://127.14.227.130:33963/ using document root <none> and password file <none>
I20260924 03:18:16.792351 27795 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:16.792557 27795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:16.793041 27795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:18:16 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219896, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:16.833348 27795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "b41a7da3e1154cb99104012a0ea642f2"
format_stamp: "Formatted at 2026-09-24 03:18:16 on dist-test-slave-vd0p"
server_key: "b87b7ae3727a8514c6d99dde900d6e50"
server_key_iv: "04f81244d0c74cd2cf940b5f93026ad8"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:16.834581 27795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "b41a7da3e1154cb99104012a0ea642f2"
format_stamp: "Formatted at 2026-09-24 03:18:16 on dist-test-slave-vd0p"
server_key: "b87b7ae3727a8514c6d99dde900d6e50"
server_key_iv: "04f81244d0c74cd2cf940b5f93026ad8"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:16.851650 27795 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.007s
I20260924 03:18:16.891788 27812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:16.894172 27795 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.008s sys 0.001s
I20260924 03:18:16.894387 27795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "b41a7da3e1154cb99104012a0ea642f2"
format_stamp: "Formatted at 2026-09-24 03:18:16 on dist-test-slave-vd0p"
server_key: "b87b7ae3727a8514c6d99dde900d6e50"
server_key_iv: "04f81244d0c74cd2cf940b5f93026ad8"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:16.894661 27795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:16.919458 27795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:16.925027 27795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:16.925424 27795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:16.926627 27795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:16.928393 27795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:16.928541 27795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:16.928704 27795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:16.928818 27795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:16.971349 27931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:43049 every 8 connection(s)
I20260924 03:18:16.971343 27795 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:43049
I20260924 03:18:16.972982 27795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:18:16.976009 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 27795
I20260924 03:18:16.976199 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:18:16.976926 15246 external_mini_cluster.cc:1428] Setting key 925150c95850af3eecf3b7f4ba27447a
Sep 24 03:18:16 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219896, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
I20260924 03:18:17.009415 27934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:16.975517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:59775 (local address 127.14.227.190:33897)
0924 03:18:16.989888 (+ 14371us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:16.989895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:16.989917 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:18:16.990168 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:16.990174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:16.990264 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:16.990381 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:16.992478 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:16.993614 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:16.996276 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:16.996881 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:16.997042 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:17.001606 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:17.001614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:17.001620 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:17.001669 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:17.005629 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.006578 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.006582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.006584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.006704 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.007272 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.007280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.007286 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.007591 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:17.007708 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:17.008714 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0924 03:18:17.008974 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14248,"thread_start_us":14130,"threads_started":1}
I20260924 03:18:17.010576 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:16.976120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33897 (local address 127.14.227.130:59775)
0924 03:18:16.986695 (+ 10575us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:16.986796 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:16.989846 (+ 3050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:16.990672 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:16.990686 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:16.991398 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:16.992268 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:16.992297 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:16.993804 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:16.993812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:16.996100 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:16.996127 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:16.997157 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:16.997168 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:16.997496 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:16.998342 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:16.998378 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:17.001348 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:17.005787 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:17.005795 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:17.005814 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:17.006418 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:17.006824 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:17.006829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:17.006833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:17.007087 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:17.007723 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:17.007731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:17.008221 (+ 490us) client_negotiation.cc:769] Sending connection context
0924 03:18:17.008571 (+ 350us) client_negotiation.cc:240] Negotiation successful
0924 03:18:17.009122 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10164,"thread_start_us":9923,"threads_started":1}
I20260924 03:18:17.012228 27932 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33897
I20260924 03:18:17.012560 27932 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:17.013447 27932 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
I20260924 03:18:17.015250 27576 ts_manager.cc:194] Registered new tserver with Master: b41a7da3e1154cb99104012a0ea642f2 (127.14.227.130:43049)
I20260924 03:18:17.016333 27576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:59775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:18:17.036686 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:52709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33897
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:18:17.210137 27789 heartbeater.cc:503] Master 127.14.227.190:33897 was elected leader, sending a full tablet report...
W20260924 03:18:17.359335 27938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:17.359628 27938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:17.359787 27938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:17.366973 27938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:17.367125 27938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:17.367285 27938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:18:17.376690 27938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.227.148:52709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:17.378094 27938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:17.379372 27938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:17.393095 27944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:17.393172 27946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:17.393846 27943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:17.394279 27938 server_base.cc:1034] running on GCE node
I20260924 03:18:17.394853 27938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:17.395771 27938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:17.396996 27938 hybrid_clock.cc:648] HybridClock initialized: now 1790219897396945 us; error 61 us; skew 500 ppm
I20260924 03:18:17.397320 27938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:17 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:17.400424 27938 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:18:17.401823 27938 webserver.cc:460] Webserver started at http://127.14.227.131:37677/ using document root <none> and password file <none>
I20260924 03:18:17.402516 27938 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:17.402612 27938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:17.402878 27938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 03:18:17 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:17.420187 27938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe"
format_stamp: "Formatted at 2026-09-24 03:18:17 on dist-test-slave-vd0p"
server_key: "46e84b1d90a2275fa876a9acf5c48c8a"
server_key_iv: "73737218e978861ddc4b53eb8dbd5c67"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:17.421258 27938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe"
format_stamp: "Formatted at 2026-09-24 03:18:17 on dist-test-slave-vd0p"
server_key: "46e84b1d90a2275fa876a9acf5c48c8a"
server_key_iv: "73737218e978861ddc4b53eb8dbd5c67"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:17.427774 27938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 03:18:17.444090 27953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:17.445719 27938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:18:17.445876 27938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe"
format_stamp: "Formatted at 2026-09-24 03:18:17 on dist-test-slave-vd0p"
server_key: "46e84b1d90a2275fa876a9acf5c48c8a"
server_key_iv: "73737218e978861ddc4b53eb8dbd5c67"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:17.446201 27938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:17.469473 27938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:17.472788 27938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:17.473032 27938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:17.473815 27938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:17.475071 27938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:17.475139 27938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:17.475219 27938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:17.475262 27938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:17.508724 27938 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:37471
I20260924 03:18:17.508755 28065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:37471 every 8 connection(s)
I20260924 03:18:17.510370 27938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:18:17.520041 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 27938
I20260924 03:18:17.520220 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:18:17.520895 15246 external_mini_cluster.cc:1428] Setting key 6cc26137ba880d75825c8386dfeea6a0
Sep 24 03:18:17 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:17.529392 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:17.512839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:51869 (local address 127.14.227.190:33897)
0924 03:18:17.513258 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:17.513262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:17.514347 (+ 1085us) server_negotiation.cc:407] Connection header received
0924 03:18:17.516056 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:17.516061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:17.516176 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:17.516289 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:17.517592 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.518449 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.520069 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.520549 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.520703 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:17.524537 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:17.524541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:17.524545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:17.524577 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:17.526942 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.527593 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.527596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.527598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.527717 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.528095 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.528098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.528100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.528345 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:17.528448 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:17.529004 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 03:18:17.529153 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":246,"threads_started":1}
I20260924 03:18:17.530372 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:17.513460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33897 (local address 127.14.227.131:51869)
0924 03:18:17.514195 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:17.514248 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:17.515810 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:17.516510 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:17.516518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:17.516890 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:17.517428 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:17.517443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.518587 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.518591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:17.519938 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:17.519949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.520767 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.520795 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:17.521105 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:17.521745 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:17.521766 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:17.524343 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:17.527074 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:17.527081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:17.527092 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:17.527501 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:17.527799 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:17.527802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:17.527805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:17.527982 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:17.528439 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:17.528445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:17.528725 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 03:18:17.528934 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 03:18:17.529311 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":308,"threads_started":1}
I20260924 03:18:17.531610 28066 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33897
I20260924 03:18:17.531862 28066 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:17.532454 28066 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
I20260924 03:18:17.533723 27576 ts_manager.cc:194] Registered new tserver with Master: 6ade7d7033c0440ba7c4ab3bcb41b7fe (127.14.227.131:37471)
I20260924 03:18:17.534605 27576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:51869
I20260924 03:18:17.536092 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:18:17.537703 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 27656
Sep 24 03:18:17 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:18:17.553474 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:17.538566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.14.227.129:36517)
0924 03:18:17.539127 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:17.539133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:17.539155 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:18:17.539208 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:17.539214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:17.539401 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:17.539521 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:17.540906 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.541831 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.543295 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.543777 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.543868 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:17.548347 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:17.548358 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:17.548370 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:17.548401 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:17.550996 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.551648 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.551654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.551656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.551793 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.552118 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.552121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.552123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.552434 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:17.552697 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:17.552979 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 03:18:17.553161 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":284,"threads_started":1}
I20260924 03:18:17.554767 27784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:17.637629 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 27656
I20260924 03:18:17.653177 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 27795
Sep 24 03:18:17 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:18:17.669040 28073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:17.654090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.14.227.130:43049)
0924 03:18:17.654592 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:17.654599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:17.654621 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:18:17.654700 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:17.654707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:17.654903 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:17.655073 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:17.656232 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.657148 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.658512 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.659044 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.659190 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:17.663434 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:17.663448 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:17.663467 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:17.663506 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:17.666109 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.666810 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.666823 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.666825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.666960 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.667322 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.667327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.667330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.667703 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:17.668062 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:17.668393 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 03:18:17.668661 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
I20260924 03:18:17.670307 27927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:17.750371 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 27795
I20260924 03:18:17.765631 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 27938
Sep 24 03:18:17 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:18:17.787050 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:17.766589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.14.227.131:37471)
0924 03:18:17.767165 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:17.767180 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:17.767207 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:18:17.767265 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:17.767271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:17.767478 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:17.767645 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:17.768951 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.770011 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.771284 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:17.771831 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:17.771968 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:17.781827 (+ 9859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:17.781839 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:17.781854 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:17.781885 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:17.784391 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.785043 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.785050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.785053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.785178 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:17.785522 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:17.785529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:17.785532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:17.785853 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:17.786218 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:17.786422 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:18:17.786702 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":306,"threads_started":1}
I20260924 03:18:17.788255 28061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:17.874752 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 27938
I20260924 03:18:17.895551 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 27544
I20260924 03:18:17.897078 27606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:18.013908 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 27544
I20260924 03:18:18.036983 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33897
--webserver_interface=127.14.227.190
--webserver_port=36779
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:33897
--ranger_config_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:52709/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.227.148:52709/kms with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:18.360307 28078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:18.360592 28078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:18.360778 28078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:18.368021 28078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:18.368127 28078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:18.368213 28078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:18.368296 28078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:18.368371 28078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:18.378435 28078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.227.148:52709/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:33897
--ranger_config_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=36779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:18.379735 28078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:18.381145 28078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:18.393294 28083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:18.393294 28084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:18.393447 28086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:18.394161 28078 server_base.cc:1034] running on GCE node
I20260924 03:18:18.394955 28078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:18.396503 28078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:18.397758 28078 hybrid_clock.cc:648] HybridClock initialized: now 1790219898397713 us; error 55 us; skew 500 ppm
I20260924 03:18:18.398217 28078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:18 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:18.401371 28078 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:18:18.403111 28078 webserver.cc:460] Webserver started at http://127.14.227.190:36779/ using document root <none> and password file <none>
I20260924 03:18:18.403818 28078 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:18.403977 28078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:18:18 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:18.455669 28078 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260924 03:18:18.458959 28094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:18.460374 28078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:18:18.460585 28078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "b6aec7a05b3147ea90adaa3d91df765e"
format_stamp: "Formatted at 2026-09-24 03:18:14 on dist-test-slave-vd0p"
server_key: "f89151b287673eac75e318ca515a4143"
server_key_iv: "038d5ef6fff469b21cbace60f8852a6b"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:18.461572 28078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:18.491120 28078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:18.495208 28078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:18.495501 28078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:18.513242 28078 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:33897
I20260924 03:18:18.513278 28145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:33897 every 8 connection(s)
I20260924 03:18:18.514763 28078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:18:18.518586 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 28078
I20260924 03:18:18.519419 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:52709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:36517
--local_ip_for_outbound_sockets=127.14.227.129
--tserver_master_addrs=127.14.227.190:33897
--webserver_port=40917
--webserver_interface=127.14.227.129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.227.148:52709/kms with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:18:18.521724 28146 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:18:18.525053 28146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Bootstrap starting.
I20260924 03:18:18.547457 28146 log.cc:824] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:18.556262 28146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:18:18.556778 28146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Bootstrap complete.
I20260924 03:18:18.565868 28146 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } }
I20260924 03:18:18.566855 28146 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6aec7a05b3147ea90adaa3d91df765e, State: Initialized, Role: FOLLOWER
I20260924 03:18:18.567476 28146 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } }
I20260924 03:18:18.567716 28146 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:18:18.567845 28146 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:18:18.568009 28146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:18:18.571779 28146 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } }
I20260924 03:18:18.572260 28146 leader_election.cc:302] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6aec7a05b3147ea90adaa3d91df765e; no voters:
I20260924 03:18:18.572906 28146 leader_election.cc:288] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:18:18.573050 28150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:18:18.574270 28150 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [term 2 LEADER]: Becoming Leader. State: Replica: b6aec7a05b3147ea90adaa3d91df765e, State: Running, Role: LEADER
I20260924 03:18:18.574882 28150 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } }
I20260924 03:18:18.575126 28146 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:18.577242 28152 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6aec7a05b3147ea90adaa3d91df765e. Latest consensus state: current_term: 2 leader_uuid: "b6aec7a05b3147ea90adaa3d91df765e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } } }
I20260924 03:18:18.577601 28152 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:18.578123 28151 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b6aec7a05b3147ea90adaa3d91df765e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aec7a05b3147ea90adaa3d91df765e" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33897 } } }
I20260924 03:18:18.578374 28151 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:18.586884 28155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:18:18.593295 28155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:18:18.595104 28155 catalog_manager.cc:1259] Loaded cluster ID: ef28dcc82f7347d5a7822fa273f6e297
I20260924 03:18:18.595275 28155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:18:18.600414 28155 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:18:18.602320 28155 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Loaded TSK: 0
I20260924 03:18:18.604290 28155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:18:18.607641 28146 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 03:18:18.971472 28148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:18.971938 28148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:18.972139 28148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:18.983807 28148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:18.983950 28148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:18.984207 28148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:18:19.000011 28148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.227.148:52709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:36517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=40917
--tserver_master_addrs=127.14.227.190:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:19.001763 28148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:19.003573 28148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:19.056738 28184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:19.064549 28183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:19.065660 28186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:19.066360 28148 server_base.cc:1034] running on GCE node
I20260924 03:18:19.067356 28148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:19.068797 28148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:19.070066 28148 hybrid_clock.cc:648] HybridClock initialized: now 1790219899069980 us; error 69 us; skew 500 ppm
I20260924 03:18:19.070474 28148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:19 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:19.074644 28148 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:18:19.101369 28148 webserver.cc:460] Webserver started at http://127.14.227.129:40917/ using document root <none> and password file <none>
I20260924 03:18:19.102696 28148 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:19.102909 28148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:18:19 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:19.141284 28148 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 03:18:19.162644 28195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:19.171247 28148 fs_manager.cc:725] Time spent opening block manager: real 0.028s user 0.013s sys 0.007s
I20260924 03:18:19.171486 28148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "60df20d8272f4943a0598d06f8e5d960"
format_stamp: "Formatted at 2026-09-24 03:18:15 on dist-test-slave-vd0p"
server_key: "8c0fc2f7394a58771476e5b8b9dfc777"
server_key_iv: "1ce401021bc8e6149bac6463ef4519f9"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:19.173010 28148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:19.213266 28148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:19.218542 28148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:19.218887 28148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:19.220129 28148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:19.221982 28148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:19.222106 28148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 03:18:19.222218 28148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:19.222306 28148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:19.281528 28148 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:36517
I20260924 03:18:19.281548 28307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:36517 every 8 connection(s)
I20260924 03:18:19.283434 28148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:18:19.293787 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 28148
I20260924 03:18:19.294624 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:52709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:43049
--local_ip_for_outbound_sockets=127.14.227.130
--tserver_master_addrs=127.14.227.190:33897
--webserver_port=33963
--webserver_interface=127.14.227.130
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.227.148:52709/kms with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:18:19 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:19.319800 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:19.286524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:59679 (local address 127.14.227.190:33897)
0924 03:18:19.287284 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:19.287304 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:19.290752 (+ 3448us) server_negotiation.cc:407] Connection header received
0924 03:18:19.293587 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:19.293607 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:19.294137 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:19.294530 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:19.296954 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:19.298998 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:19.303203 (+ 4205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:19.304302 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:19.304469 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:19.309202 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:19.309214 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:19.309236 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:19.309290 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:19.313582 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:19.314701 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:19.314713 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:19.314724 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:19.314922 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:19.315504 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:19.315512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:19.315516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:19.316250 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:19.316446 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:19.317646 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0924 03:18:19.318184 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":599,"threads_started":1}
I20260924 03:18:19.322400 28311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:19.286652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33897 (local address 127.14.227.129:59679)
0924 03:18:19.290548 (+ 3896us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:19.290596 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:19.292866 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:19.295271 (+ 2405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:19.295285 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:19.295998 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:19.296745 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:19.296770 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:19.299221 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:19.299229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:19.303034 (+ 3805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:19.303052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:19.304924 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:19.304933 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:19.305243 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:19.306078 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:19.306110 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:19.308929 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:19.313814 (+ 4885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:19.313826 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:19.313846 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:19.314515 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:19.315084 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:19.315091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:19.315097 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:19.315362 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:19.316560 (+ 1198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:19.316571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:19.317030 (+ 459us) client_negotiation.cc:769] Sending connection context
0924 03:18:19.320430 (+ 3400us) client_negotiation.cc:240] Negotiation successful
0924 03:18:19.321050 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3592,"thread_start_us":3406,"threads_started":1}
I20260924 03:18:19.325934 28308 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33897
I20260924 03:18:19.326247 28308 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:19.327057 28308 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
W20260924 03:18:19.330101 28308 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:33897 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:18:19.330302 28308 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:19.330540 28308 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
I20260924 03:18:19.331544 28308 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:19.331781 28308 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
W20260924 03:18:19.332717 28308 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:18:19.733026 28312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:19.733448 28312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:19.733611 28312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:19.745139 28312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:19.745272 28312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:19.745466 28312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:18:19.760918 28312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.227.148:52709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:43049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=33963
--tserver_master_addrs=127.14.227.190:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:19.762614 28312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:19.764222 28312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 24 03:18:19 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 03:18:19.782816 28329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:19.783855 28328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:19.791664 28331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:19.791827 28312 server_base.cc:1034] running on GCE node
I20260924 03:18:19.792755 28312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:19.794099 28312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:19.795368 28312 hybrid_clock.cc:648] HybridClock initialized: now 1790219899795301 us; error 57 us; skew 500 ppm
I20260924 03:18:19.795771 28312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:19 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:19.804069 28312 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:18:19.812284 28312 webserver.cc:460] Webserver started at http://127.14.227.130:33963/ using document root <none> and password file <none>
I20260924 03:18:19.813225 28312 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:19.813373 28312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:18:19 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:19.843716 28312 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.005s sys 0.001s
I20260924 03:18:19.847548 28340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:19.849347 28312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:18:19.849534 28312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "b41a7da3e1154cb99104012a0ea642f2"
format_stamp: "Formatted at 2026-09-24 03:18:16 on dist-test-slave-vd0p"
server_key: "b87b7ae3727a8514c6d99dde900d6e50"
server_key_iv: "04f81244d0c74cd2cf940b5f93026ad8"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:19.850947 28312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:19.874907 28312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:19.879077 28312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:19.879415 28312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:19.882884 28312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:19.884593 28312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:19.884713 28312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:19.884824 28312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:19.884900 28312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:19.955478 28312 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:43049
I20260924 03:18:19.955550 28452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:43049 every 8 connection(s)
I20260924 03:18:19.957438 28312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:18:19.965344 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 28312
I20260924 03:18:19.966102 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.227.148:52709/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:37471
--local_ip_for_outbound_sockets=127.14.227.131
--tserver_master_addrs=127.14.227.190:33897
--webserver_port=37677
--webserver_interface=127.14.227.131
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.227.148:52709/kms with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:18:19 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:20.008335 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:19.961492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:50209 (local address 127.14.227.190:33897)
0924 03:18:19.984388 (+ 22896us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:19.984397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:19.985658 (+ 1261us) server_negotiation.cc:407] Connection header received
0924 03:18:19.989286 (+ 3628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:19.989293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:19.989389 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:19.989476 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:19.991797 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:19.992973 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:19.995406 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:19.996060 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:19.996156 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:20.000888 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:20.000895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:20.000900 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:20.000941 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:20.004589 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:20.005561 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:20.005567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:20.005571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:20.005713 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:20.006277 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:20.006282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:20.006286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:20.006639 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:20.006767 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:20.007733 (+ 966us) server_negotiation.cc:299] Negotiation successful
0924 03:18:20.007955 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22721,"thread_start_us":22925,"threads_started":1}
I20260924 03:18:20.010848 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:19.960921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33897 (local address 127.14.227.130:50209)
0924 03:18:19.985119 (+ 24198us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:19.985189 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:19.988031 (+ 2842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:19.990103 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:19.990116 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:19.990761 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:19.991587 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:19.991613 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:19.993169 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:19.993175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:19.995232 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:19.995246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:19.996500 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:19.996508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:19.996779 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:19.997592 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:19.997624 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:20.000587 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:20.004779 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:20.004788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:20.004806 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:20.005407 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:20.005853 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:20.005857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:20.005861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:20.006142 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:20.006863 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:20.006872 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:20.007294 (+ 422us) client_negotiation.cc:769] Sending connection context
0924 03:18:20.008892 (+ 1598us) client_negotiation.cc:240] Negotiation successful
0924 03:18:20.009450 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23846,"thread_start_us":27343,"threads_started":1}
I20260924 03:18:20.012846 28453 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33897
I20260924 03:18:20.013162 28453 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:20.014070 28453 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
I20260924 03:18:20.016228 28111 ts_manager.cc:194] Registered new tserver with Master: b41a7da3e1154cb99104012a0ea642f2 (127.14.227.130:43049)
I20260924 03:18:20.018421 28111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:50209
Sep 24 03:18:20 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:20.332868 28308 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:20.333282 28308 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
I20260924 03:18:20.334496 28111 ts_manager.cc:194] Registered new tserver with Master: 60df20d8272f4943a0598d06f8e5d960 (127.14.227.129:36517)
I20260924 03:18:20.335722 28111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:59679
W20260924 03:18:20.445565 28457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:20.446091 28457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:20.446296 28457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:20.457715 28457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:20.457932 28457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:20.458189 28457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:18:20.473465 28457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.227.148:52709/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:37471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=37677
--tserver_master_addrs=127.14.227.190:33897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:20.475349 28457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:20.477030 28457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 24 03:18:20 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
W20260924 03:18:20.511415 28463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:20.519042 28464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:20.529350 28457 server_base.cc:1034] running on GCE node
W20260924 03:18:20.530468 28467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:20.531428 28457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:20.537722 28457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:20.538995 28457 hybrid_clock.cc:648] HybridClock initialized: now 1790219900538913 us; error 67 us; skew 500 ppm
I20260924 03:18:20.539438 28457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:20 dist-test-slave-vd0p krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:20.543716 28457 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:18:20.563378 28457 webserver.cc:460] Webserver started at http://127.14.227.131:37677/ using document root <none> and password file <none>
I20260924 03:18:20.564486 28457 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:20.564766 28457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 03:18:20 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for HTTP/127.14.227.148@KRBTEST.COM
I20260924 03:18:20.631915 28457 fs_manager.cc:709] Time spent opening directory manager: real 0.039s user 0.009s sys 0.025s
I20260924 03:18:20.639029 28474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:20.645358 28457 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260924 03:18:20.645661 28457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe"
format_stamp: "Formatted at 2026-09-24 03:18:17 on dist-test-slave-vd0p"
server_key: "46e84b1d90a2275fa876a9acf5c48c8a"
server_key_iv: "73737218e978861ddc4b53eb8dbd5c67"
server_key_version: "kuduclusterkey@0"
I20260924 03:18:20.647516 28457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:20.712244 28457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:20.719848 28457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:20.720224 28457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:20.727653 28457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:20.735812 28457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:20.736032 28457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 03:18:20.736199 28457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:20.736318 28457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:20.888568 28457 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:37471
I20260924 03:18:20.888646 28593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:37471 every 8 connection(s)
I20260924 03:18:20.890437 28457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:18:20.891067 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 28457
I20260924 03:18:20.956292 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:20.892031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.14.227.190:33897)
0924 03:18:20.922410 (+ 30379us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:20.922420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:20.922457 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:18:20.922539 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:20.922548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:20.922667 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:20.922818 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:20.926085 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:20.927203 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:20.930406 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:20.931069 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:20.931192 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:20.938750 (+ 7558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:20.938758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:20.938762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:20.938822 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:20.942262 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:20.947543 (+ 5281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:20.947556 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:20.947560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:20.947730 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:20.954340 (+ 6610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:20.954345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:20.954348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:20.954734 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:20.955035 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:20.955630 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 03:18:20.955907 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30207,"thread_start_us":30091,"threads_started":1}
Sep 24 03:18:20 dist-test-slave-vd0p krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:20.975768 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:20.917722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:39063 (local address 127.14.227.190:33897)
0924 03:18:20.948264 (+ 30542us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:20.948270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:20.952836 (+ 4566us) server_negotiation.cc:407] Connection header received
0924 03:18:20.952896 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:20.952901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:20.953000 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:20.953088 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:20.959258 (+ 6170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:20.960375 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:20.962575 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:20.963140 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:20.963225 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:20.967669 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:20.967676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:20.967680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:20.967720 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:20.971485 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:20.973147 (+ 1662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:20.973153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:20.973156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:20.973290 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:20.973777 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:20.973782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:20.973786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:20.974132 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:20.974253 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:20.975197 (+ 944us) server_negotiation.cc:299] Negotiation successful
0924 03:18:20.975413 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30413,"thread_start_us":30532,"threads_started":1}
I20260924 03:18:20.978243 28598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:20.917065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33897 (local address 127.14.227.131:39063)
0924 03:18:20.949778 (+ 32713us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:20.949845 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:20.952400 (+ 2555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:20.956972 (+ 4572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:20.956986 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:20.957987 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:20.958825 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:20.958850 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:20.960566 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:20.960571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:20.962414 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:20.962428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:20.963553 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:20.963559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:20.963849 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:20.964596 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:20.964623 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:20.967423 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:20.972124 (+ 4701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:20.972133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:20.972149 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:20.973021 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:20.973416 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:20.973421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:20.973424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:20.973657 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:20.974353 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:20.974361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:20.974750 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 03:18:20.976300 (+ 1550us) client_negotiation.cc:240] Negotiation successful
0924 03:18:20.976831 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32351,"thread_start_us":36061,"threads_started":1}
I20260924 03:18:20.980083 28594 heartbeater.cc:347] Connected to a master server at 127.14.227.190:33897
I20260924 03:18:20.980365 28594 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:20.981140 28594 heartbeater.cc:511] Master 127.14.227.190:33897 requested a full tablet report, sending...
I20260924 03:18:20.982703 28111 ts_manager.cc:194] Registered new tserver with Master: 6ade7d7033c0440ba7c4ab3bcb41b7fe (127.14.227.131:37471)
I20260924 03:18:20.983712 28111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:39063
I20260924 03:18:20.988678 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:18:21.023399 28453 heartbeater.cc:503] Master 127.14.227.190:33897 was elected leader, sending a full tablet report...
I20260924 03:18:21.067239 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.037693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.14.227.190:33897)
0924 03:18:21.037940 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.037973 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.050580 (+ 12607us) server_negotiation.cc:407] Connection header received
0924 03:18:21.050815 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.050822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.050917 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.051005 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:21.053274 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.054355 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.056317 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.056913 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.057026 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.060429 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:21.060435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:21.060440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:21.060480 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:21.063912 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:21.064983 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:21.064989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:21.064992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:21.065135 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:21.065533 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:21.065538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:21.065542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:21.065905 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:21.066124 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.066706 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.066915 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:18:21.073972 28111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:18:21.085083 28111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:18:21.148505 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.112363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:43049 (local address 127.0.0.1:33376)
0924 03:18:21.116366 (+ 4003us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:21.116422 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:21.116650 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:21.132997 (+ 16347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:21.133007 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:21.133049 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:21.133589 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.133607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.141340 (+ 7733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.141348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.144356 (+ 3008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.144374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.147018 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.147029 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.147255 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.147309 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 03:18:21.147446 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 03:18:21.147657 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3635,"mutex_wait_us":1,"thread_start_us":17384,"threads_started":1}
I20260924 03:18:21.149766 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.112745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.14.227.130:43049)
0924 03:18:21.132024 (+ 19279us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.132041 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.132091 (+ 50us) server_negotiation.cc:407] Connection header received
0924 03:18:21.132254 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.132268 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.132683 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.132968 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:21.138688 (+ 5720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.141176 (+ 2488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.144539 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.146418 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.146607 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.146838 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.149232 (+ 2394us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.149406 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19002,"thread_start_us":20438,"threads_started":1}
I20260924 03:18:21.154366 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.130292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:37471 (local address 127.0.0.1:50968)
0924 03:18:21.135539 (+ 5247us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:21.135565 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:21.135706 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:21.146558 (+ 10852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:21.146588 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:21.146614 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:21.147046 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.147057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.149503 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.149509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.151620 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.151636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.153524 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.153530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.153736 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.153767 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:18:21.153834 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:18:21.153976 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5086,"mutex_wait_us":2,"thread_start_us":5505,"threads_started":1}
I20260924 03:18:21.154083 28388 tablet_service.cc:1467] Processing CreateTablet for tablet 958e70317a094053948fb51c5d361993 (DEFAULT_TABLE table=test-table [id=39610143f5c7472382487fc42e00fe7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:21.155187 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.115064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:36517 (local address 127.0.0.1:45892)
0924 03:18:21.129377 (+ 14313us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:21.129412 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:21.129615 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:21.139059 (+ 9444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:21.139067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:21.139101 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:21.139495 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.139508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.141529 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.141536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.151496 (+ 9960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.151515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.154265 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.154274 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.154516 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.154564 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 03:18:21.154662 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:18:21.154799 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14125,"mutex_wait_us":7293,"thread_start_us":777,"threads_started":1}
I20260924 03:18:21.155705 28388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958e70317a094053948fb51c5d361993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:21.155809 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.130862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.14.227.131:37471)
0924 03:18:21.145685 (+ 14823us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.145695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.145728 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:18:21.145830 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.145839 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.146134 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.147261 (+ 1127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:21.147573 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.149358 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.151849 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.153298 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.155060 (+ 1762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.155238 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.155309 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.155462 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14626,"thread_start_us":15397,"threads_started":1}
I20260924 03:18:21.156425 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.115409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.14.227.129:36517)
0924 03:18:21.137958 (+ 22549us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.137970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.138003 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:18:21.138167 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.138178 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.138489 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.138713 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:21.139703 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.141328 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.151706 (+ 10378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.153583 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.153825 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.154063 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.155707 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.155920 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22292,"thread_start_us":24992,"threads_started":1}
I20260924 03:18:21.159701 28529 tablet_service.cc:1467] Processing CreateTablet for tablet 958e70317a094053948fb51c5d361993 (DEFAULT_TABLE table=test-table [id=39610143f5c7472382487fc42e00fe7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:21.161222 28529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958e70317a094053948fb51c5d361993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:21.160611 28243 tablet_service.cc:1467] Processing CreateTablet for tablet 958e70317a094053948fb51c5d361993 (DEFAULT_TABLE table=test-table [id=39610143f5c7472382487fc42e00fe7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:21.162536 28243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958e70317a094053948fb51c5d361993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:21.192512 28618 tablet_bootstrap.cc:492] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Bootstrap starting.
I20260924 03:18:21.195111 28617 tablet_bootstrap.cc:492] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Bootstrap starting.
I20260924 03:18:21.198725 28616 tablet_bootstrap.cc:492] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Bootstrap starting.
I20260924 03:18:21.201494 28618 tablet_bootstrap.cc:654] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:21.204300 28616 tablet_bootstrap.cc:654] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:21.205322 28617 tablet_bootstrap.cc:654] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:21.205847 28616 log.cc:824] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:21.208156 28618 log.cc:824] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:21.209438 28617 log.cc:824] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:21.221280 28616 tablet_bootstrap.cc:492] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: No bootstrap required, opened a new log
I20260924 03:18:21.221716 28616 ts_tablet_manager.cc:1397] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.004s
I20260924 03:18:21.223513 28617 tablet_bootstrap.cc:492] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: No bootstrap required, opened a new log
I20260924 03:18:21.223878 28617 ts_tablet_manager.cc:1397] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.006s
I20260924 03:18:21.221937 28618 tablet_bootstrap.cc:492] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: No bootstrap required, opened a new log
I20260924 03:18:21.224262 28618 ts_tablet_manager.cc:1397] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.007s
I20260924 03:18:21.239881 28617 raft_consensus.cc:348] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.245846 28618 raft_consensus.cc:348] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.246510 28617 raft_consensus.cc:374] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:21.246675 28617 raft_consensus.cc:729] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41a7da3e1154cb99104012a0ea642f2, State: Initialized, Role: FOLLOWER
I20260924 03:18:21.247159 28616 raft_consensus.cc:348] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.247520 28617 consensus_queue.cc:260] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.247776 28616 raft_consensus.cc:374] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:21.247900 28616 raft_consensus.cc:729] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ade7d7033c0440ba7c4ab3bcb41b7fe, State: Initialized, Role: FOLLOWER
I20260924 03:18:21.247985 28618 raft_consensus.cc:374] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:21.248143 28618 raft_consensus.cc:729] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60df20d8272f4943a0598d06f8e5d960, State: Initialized, Role: FOLLOWER
I20260924 03:18:21.248664 28616 consensus_queue.cc:260] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.248812 28618 consensus_queue.cc:260] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.259708 28618 ts_tablet_manager.cc:1428] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Time spent starting tablet: real 0.034s user 0.014s sys 0.002s
I20260924 03:18:21.260041 28308 heartbeater.cc:503] Master 127.14.227.190:33897 was elected leader, sending a full tablet report...
I20260924 03:18:21.260380 28594 heartbeater.cc:503] Master 127.14.227.190:33897 was elected leader, sending a full tablet report...
I20260924 03:18:21.261325 28616 ts_tablet_manager.cc:1428] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Time spent starting tablet: real 0.039s user 0.015s sys 0.004s
I20260924 03:18:21.266886 28617 ts_tablet_manager.cc:1428] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Time spent starting tablet: real 0.042s user 0.004s sys 0.013s
W20260924 03:18:21.286286 28309 tablet.cc:2369] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:18:21.430601 28624 raft_consensus.cc:482] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:21.430917 28624 raft_consensus.cc:504] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.432607 28624 leader_election.cc:288] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b41a7da3e1154cb99104012a0ea642f2 (127.14.227.130:43049), 60df20d8272f4943a0598d06f8e5d960 (127.14.227.129:36517)
I20260924 03:18:21.438334 28598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.432873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:43049 (local address 127.14.227.131:45599)
0924 03:18:21.433201 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:21.433222 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:21.433358 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:21.433772 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:21.433776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:21.433805 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:21.434096 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.434108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.435158 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.435163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.436509 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.436520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.437598 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.437605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.437784 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.437807 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:18:21.437915 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:18:21.438082 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260924 03:18:21.438334 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.433192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:45599 (local address 127.14.227.130:43049)
0924 03:18:21.433373 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.433378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.433395 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:18:21.433537 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.433542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.433632 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.433750 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:21.434241 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.434988 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.436676 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.437448 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.437624 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.437700 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.437940 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.438106 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:18:21.439486 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.433508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:36517 (local address 127.14.227.131:51941)
0924 03:18:21.434397 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:21.434412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:21.434547 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:21.434864 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:21.434868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:21.434891 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:21.435314 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.435323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.436346 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.436353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.437644 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:21.437654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.438978 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.438982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:21.439107 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.439127 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 03:18:21.439196 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:18:21.439273 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":560,"threads_started":1}
I20260924 03:18:21.439630 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.433046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:51941 (local address 127.14.227.129:36517)
0924 03:18:21.433225 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.433232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.434507 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 03:18:21.434687 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.434693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.434755 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.434848 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:21.435467 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.436207 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.437798 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.438783 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.438987 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.439061 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.439242 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.439372 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:18:21.439426 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958e70317a094053948fb51c5d361993" candidate_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41a7da3e1154cb99104012a0ea642f2" is_pre_election: true
I20260924 03:18:21.440025 28408 raft_consensus.cc:2454] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ade7d7033c0440ba7c4ab3bcb41b7fe in term 0.
I20260924 03:18:21.440640 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958e70317a094053948fb51c5d361993" candidate_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60df20d8272f4943a0598d06f8e5d960" is_pre_election: true
I20260924 03:18:21.440748 28475 leader_election.cc:302] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ade7d7033c0440ba7c4ab3bcb41b7fe, b41a7da3e1154cb99104012a0ea642f2; no voters:
I20260924 03:18:21.441135 28624 raft_consensus.cc:2781] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:18:21.441150 28263 raft_consensus.cc:2454] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ade7d7033c0440ba7c4ab3bcb41b7fe in term 0.
I20260924 03:18:21.441300 28624 raft_consensus.cc:482] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:18:21.441401 28624 raft_consensus.cc:3037] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:21.443961 28624 raft_consensus.cc:504] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.444473 28624 leader_election.cc:288] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [CANDIDATE]: Term 1 election: Requested vote from peers b41a7da3e1154cb99104012a0ea642f2 (127.14.227.130:43049), 60df20d8272f4943a0598d06f8e5d960 (127.14.227.129:36517)
I20260924 03:18:21.444926 28263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958e70317a094053948fb51c5d361993" candidate_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60df20d8272f4943a0598d06f8e5d960"
I20260924 03:18:21.444964 28408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958e70317a094053948fb51c5d361993" candidate_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41a7da3e1154cb99104012a0ea642f2"
I20260924 03:18:21.445107 28263 raft_consensus.cc:3037] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:21.445170 28408 raft_consensus.cc:3037] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:21.447300 28263 raft_consensus.cc:2454] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ade7d7033c0440ba7c4ab3bcb41b7fe in term 1.
I20260924 03:18:21.447326 28408 raft_consensus.cc:2454] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ade7d7033c0440ba7c4ab3bcb41b7fe in term 1.
I20260924 03:18:21.447758 28475 leader_election.cc:302] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ade7d7033c0440ba7c4ab3bcb41b7fe, b41a7da3e1154cb99104012a0ea642f2; no voters:
I20260924 03:18:21.448021 28624 raft_consensus.cc:2781] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:21.448473 28624 raft_consensus.cc:686] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 1 LEADER]: Becoming Leader. State: Replica: 6ade7d7033c0440ba7c4ab3bcb41b7fe, State: Running, Role: LEADER
I20260924 03:18:21.448936 28624 consensus_queue.cc:237] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } }
I20260924 03:18:21.453778 28111 catalog_manager.cc:5495] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ade7d7033c0440ba7c4ab3bcb41b7fe (127.14.227.131). New cstate: current_term: 1 leader_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ade7d7033c0440ba7c4ab3bcb41b7fe" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37471 } health_report { overall_health: HEALTHY } } }
W20260924 03:18:21.455452 28597 tablet.cc:2369] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:18:21.465852 28454 tablet.cc:2369] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:18:21.516013 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.510836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.14.227.131:37471)
0924 03:18:21.511048 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.511056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.511076 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:18:21.511337 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.511341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.511414 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.511556 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:18:21.511916 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.512592 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.513991 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.514374 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.514497 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.514978 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:18:21.515465 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:18:21.515626 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.515675 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.515749 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:18:21.524542 28263 raft_consensus.cc:1261] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 1 FOLLOWER]: Refusing update from remote peer 6ade7d7033c0440ba7c4ab3bcb41b7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:21.524581 28408 raft_consensus.cc:1261] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 1 FOLLOWER]: Refusing update from remote peer 6ade7d7033c0440ba7c4ab3bcb41b7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:21.525511 28626 consensus_queue.cc:1035] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "60df20d8272f4943a0598d06f8e5d960" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 36517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:21.525926 28624 consensus_queue.cc:1035] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41a7da3e1154cb99104012a0ea642f2" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:21.537708 28631 mvcc.cc:205] Tried to move back new op lower bound from 7332740716634062848 to 7332740716338769920. Current Snapshot: MvccSnapshot[applied={T|T < 7332740716634062848}]
I20260924 03:18:21.542846 28630 mvcc.cc:205] Tried to move back new op lower bound from 7332740716634062848 to 7332740716338769920. Current Snapshot: MvccSnapshot[applied={T|T < 7332740716634062848}]
I20260924 03:18:21.553934 28110 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:18:21.554294 28110 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:18:21.558096 28110 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6aec7a05b3147ea90adaa3d91df765e: Sending DeleteTablet for 3 replicas of tablet 958e70317a094053948fb51c5d361993
I20260924 03:18:21.559314 28388 tablet_service.cc:1514] Processing DeleteTablet for tablet 958e70317a094053948fb51c5d361993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:21 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:33376
I20260924 03:18:21.559614 28529 tablet_service.cc:1514] Processing DeleteTablet for tablet 958e70317a094053948fb51c5d361993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:21 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:50968
I20260924 03:18:21.560020 28243 tablet_service.cc:1514] Processing DeleteTablet for tablet 958e70317a094053948fb51c5d361993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:21 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:45892
I20260924 03:18:21.565589 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 28148
I20260924 03:18:21.570572 28639 tablet_replica.cc:330] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: stopping tablet replica
I20260924 03:18:21.571199 28639 raft_consensus.cc:2229] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:18:21.572237 28639 raft_consensus.cc:2258] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:21.573036 28638 tablet_replica.cc:330] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: stopping tablet replica
I20260924 03:18:21.573750 28640 tablet_replica.cc:330] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: stopping tablet replica
I20260924 03:18:21.577075 28638 raft_consensus.cc:2229] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:18:21.577541 28638 raft_consensus.cc:2258] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:21.577800 28639 ts_tablet_manager.cc:1905] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:21.578177 28640 raft_consensus.cc:2229] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:18:21.578694 28640 raft_consensus.cc:2258] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:21.580120 28638 ts_tablet_manager.cc:1905] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:21.581471 28640 ts_tablet_manager.cc:1905] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:21.585227 28639 ts_tablet_manager.cc:1918] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:21.585465 28639 log.cc:1196] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/958e70317a094053948fb51c5d361993
I20260924 03:18:21.586145 28639 ts_tablet_manager.cc:1939] T 958e70317a094053948fb51c5d361993 P 6ade7d7033c0440ba7c4ab3bcb41b7fe: Deleting consensus metadata
I20260924 03:18:21.586760 28638 ts_tablet_manager.cc:1918] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:21.586977 28638 log.cc:1196] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/958e70317a094053948fb51c5d361993
I20260924 03:18:21.587559 28638 ts_tablet_manager.cc:1939] T 958e70317a094053948fb51c5d361993 P b41a7da3e1154cb99104012a0ea642f2: Deleting consensus metadata
I20260924 03:18:21.588018 28095 catalog_manager.cc:4841] TS 6ade7d7033c0440ba7c4ab3bcb41b7fe (127.14.227.131:37471): tablet 958e70317a094053948fb51c5d361993 (table test-table [id=39610143f5c7472382487fc42e00fe7c]) successfully deleted
I20260924 03:18:21.589222 28095 catalog_manager.cc:4841] TS b41a7da3e1154cb99104012a0ea642f2 (127.14.227.130:43049): tablet 958e70317a094053948fb51c5d361993 (table test-table [id=39610143f5c7472382487fc42e00fe7c]) successfully deleted
I20260924 03:18:21.592801 28640 ts_tablet_manager.cc:1918] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:21.592957 28640 log.cc:1196] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/958e70317a094053948fb51c5d361993
I20260924 03:18:21.593443 28640 ts_tablet_manager.cc:1939] T 958e70317a094053948fb51c5d361993 P 60df20d8272f4943a0598d06f8e5d960: Deleting consensus metadata
I20260924 03:18:21.595140 28097 catalog_manager.cc:4841] TS 60df20d8272f4943a0598d06f8e5d960 (127.14.227.129:36517): tablet 958e70317a094053948fb51c5d361993 (table test-table [id=39610143f5c7472382487fc42e00fe7c]) successfully deleted
I20260924 03:18:21.598861 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.568904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.14.227.129:36517)
0924 03:18:21.569149 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.569159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.574702 (+ 5543us) server_negotiation.cc:407] Connection header received
0924 03:18:21.575042 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.575048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.575150 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.575225 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:21.581678 (+ 6453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.582789 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.587621 (+ 4832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.588231 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.588322 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.591906 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:21.591919 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:21.591938 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:21.591994 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:21.596098 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:21.596751 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:21.596760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:21.596764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:21.596918 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:21.597291 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:21.597296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:21.597299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:21.597698 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:21.598037 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.598291 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.598502 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 03:18:21.599561 28303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:21.688894 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 28148
I20260924 03:18:21.705891 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 28312
I20260924 03:18:21.719981 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.706870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.14.227.130:43049)
0924 03:18:21.707063 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.707070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.707090 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:18:21.707187 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.707192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.707262 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.707383 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:21.708650 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.709476 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.710854 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.711270 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.711392 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.713995 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:21.714016 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:21.714036 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:21.714075 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:21.717063 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:21.717723 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:21.717730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:21.717732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:21.717857 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:21.718437 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:21.718444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:21.718447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:21.718800 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:21.719132 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.719407 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.719642 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:18:21.720909 28448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:21.809108 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 28312
I20260924 03:18:21.825191 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 28457
I20260924 03:18:21.842326 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:21.826099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.14.227.131:37471)
0924 03:18:21.826305 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:21.826313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:21.826355 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:18:21.826441 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:21.826448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:21.826535 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:21.826672 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:21.827917 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.829339 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.830980 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:21.831728 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:21.831894 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:21.834694 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:21.834710 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:21.834732 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:21.834783 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:21.839156 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:21.839885 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:21.839901 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:21.839907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:21.840118 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:21.840530 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:21.840536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:21.840540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:21.841037 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:21.841415 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:21.841608 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 03:18:21.841855 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:18:21.843174 28589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:21.963918 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 28457
I20260924 03:18:21.983179 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 28078
I20260924 03:18:21.984259 28141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:22.094364 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 28078
I20260924 03:18:22.115872 15246 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 03:18:22.450521 15246 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 03:18:22.450711 15246 mini_ranger.cc:66] Stopping Ranger...
I20260924 03:18:22.795465 15246 mini_ranger.cc:68] Stopped Ranger
2026-09-24 03:18:22.795 UTC [27130] LOG: received smart shutdown request
2026-09-24 03:18:22.798 UTC [27130] LOG: background worker "logical replication launcher" (PID 27139) exited with exit code 1
2026-09-24 03:18:22.799 UTC [27134] LOG: shutting down
2026-09-24 03:18:22.956 UTC [27130] LOG: database system is shut down
2026-09-24T03:18:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52548 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:18:23 dist-test-slave-vd0p krb5kdc[28654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:18:23 dist-test-slave-vd0p krb5kdc[28654](info): set up 2 sockets
Sep 24 03:18:23 dist-test-slave-vd0p krb5kdc[28654](info): commencing operation
krb5kdc: starting...
W20260924 03:18:24.206300 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:18:24 dist-test-slave-vd0p krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:18:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:18:24.430363 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33455
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:33455,127.14.227.189:35839,127.14.227.188:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:24.745819 28667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:24.746152 28667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:24.746310 28667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:24.753235 28667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:24.753373 28667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:24.753475 28667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:24.753571 28667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:24.753643 28667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:24.763049 28667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:33455,127.14.227.189:35839,127.14.227.188:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:24.764343 28667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:24.765656 28667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:24.777122 28673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:24.777191 28672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:24.777172 28675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:24.777839 28667 server_base.cc:1034] running on GCE node
I20260924 03:18:24.778579 28667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:24.779907 28667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:24.781134 28667 hybrid_clock.cc:648] HybridClock initialized: now 1790219904781082 us; error 43 us; skew 500 ppm
I20260924 03:18:24.781481 28667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:24 dist-test-slave-vd0p krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:24.784533 28667 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:18:24.785921 28667 webserver.cc:460] Webserver started at http://127.14.227.190:41289/ using document root <none> and password file <none>
I20260924 03:18:24.786649 28667 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:24.786788 28667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:24.787083 28667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:24.789920 28667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "4d3c7f5dc253401c9ccb5758e651e57f"
format_stamp: "Formatted at 2026-09-24 03:18:24 on dist-test-slave-vd0p"
server_key: "61de92a78a677f06a918490e50e041ee"
server_key_iv: "f7fa31a96e6575d3d18a640a2ca5c82b"
server_key_version: "encryptionkey@0"
I20260924 03:18:24.790773 28667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "4d3c7f5dc253401c9ccb5758e651e57f"
format_stamp: "Formatted at 2026-09-24 03:18:24 on dist-test-slave-vd0p"
server_key: "61de92a78a677f06a918490e50e041ee"
server_key_iv: "f7fa31a96e6575d3d18a640a2ca5c82b"
server_key_version: "encryptionkey@0"
I20260924 03:18:24.795707 28667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:18:24.799041 28682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:24.800305 28667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:18:24.800480 28667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "4d3c7f5dc253401c9ccb5758e651e57f"
format_stamp: "Formatted at 2026-09-24 03:18:24 on dist-test-slave-vd0p"
server_key: "61de92a78a677f06a918490e50e041ee"
server_key_iv: "f7fa31a96e6575d3d18a640a2ca5c82b"
server_key_version: "encryptionkey@0"
I20260924 03:18:24.800661 28667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:24.823768 28667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:24.827575 28667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:24.827848 28667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:24.845992 28667 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:33455
I20260924 03:18:24.846067 28733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:33455 every 8 connection(s)
I20260924 03:18:24.847289 28667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:18:24.850737 28734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:24.851887 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 28667
I20260924 03:18:24.852111 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:18:24.852958 15246 external_mini_cluster.cc:1428] Setting key 4bf4b88da04d552c833263247aca6bc4
I20260924 03:18:24.860451 28734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.227.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.189@KRBTEST.COM" created.
Sep 24 03:18:24 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:24.885262 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:24.863742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33455 (local address 127.0.0.1:51892)
0924 03:18:24.864796 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:24.864833 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:24.866552 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:24.867315 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:24.867322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:24.867987 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:24.868291 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:24.868305 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:24.869783 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:24.869792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:24.870918 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:24.870929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:24.871774 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:24.871780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:24.871945 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:24.872546 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:24.872567 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:24.877241 (+ 4674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:24.881196 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:24.881207 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:24.881219 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:24.881600 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:24.882007 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:24.882014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:24.882017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:24.882169 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:24.882889 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:24.882896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:24.883099 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 03:18:24.883326 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 03:18:24.883934 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":168,"thread_start_us":674,"threads_started":1}
I20260924 03:18:24.885284 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:24.863729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.14.227.190:33455)
0924 03:18:24.864513 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:24.864541 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:24.864938 (+ 397us) server_negotiation.cc:407] Connection header received
0924 03:18:24.866897 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:24.866903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:24.867113 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:24.867300 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:24.868470 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:24.869630 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:24.871059 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:24.871567 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:24.871746 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:24.877548 (+ 5802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:24.877594 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:24.877608 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:24.877641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:24.881074 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:24.881723 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:24.881731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:24.881734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:24.881877 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:24.882276 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:24.882280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:24.882284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:24.882758 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:24.882958 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:24.883444 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 03:18:24.883832 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":547,"thread_start_us":363,"threads_started":1}
I20260924 03:18:24.888412 28734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:18:24.889487 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:24.888990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47500)
0924 03:18:24.889185 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:24.889314 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 03:18:24.889726 28683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
I20260924 03:18:24.891451 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:24.890994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47508)
0924 03:18:24.891186 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:24.891276 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:18:24.891645 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:35839: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab.
Entry for principal kudu/127.14.227.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab.
I20260924 03:18:24.903204 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.189:35839
--webserver_interface=127.14.227.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab
--principal=kudu/127.14.227.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:33455,127.14.227.189:35839,127.14.227.188:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:18:24.941001 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } attempt: 1
I20260924 03:18:24.942312 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:24.941712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47518)
0924 03:18:24.942003 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:24.942144 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:18:24.943317 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:24.942922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47530)
0924 03:18:24.943073 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:24.943158 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 03:18:24.943497 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:35839: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
I20260924 03:18:25.011762 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } attempt: 2
I20260924 03:18:25.013060 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.012472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47536)
0924 03:18:25.012747 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.012874 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:18:25.014211 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.013739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47540)
0924 03:18:25.013924 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.014049 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 03:18:25.014472 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:35839: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
I20260924 03:18:25.105734 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } attempt: 3
I20260924 03:18:25.106966 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.106440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47552)
0924 03:18:25.106659 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.106797 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 03:18:25.108111 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.107591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47564)
0924 03:18:25.107761 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.107921 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:18:25.108311 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:35839: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
W20260924 03:18:25.225852 28741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:25.226276 28741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:25.226444 28741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:25.233866 28741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:25.234053 28741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:25.234154 28741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:25.234264 28741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:25.234350 28741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:25.244381 28741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:33455,127.14.227.189:35839,127.14.227.188:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab
--principal=kudu/127.14.227.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.189:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.227.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:25.245824 28741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:25.247191 28741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:18:25.251543 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } attempt: 4
I20260924 03:18:25.253067 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.252244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47576)
0924 03:18:25.252732 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.252847 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260924 03:18:25.254433 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.253821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47592)
0924 03:18:25.254039 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.254224 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 03:18:25.254729 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:35839: Network error: Client connection negotiation failed: client connection to 127.14.227.189:35839: connect: Connection refused (error 111)
W20260924 03:18:25.259115 28749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:25.259076 28747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:25.259158 28746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:25.260103 28741 server_base.cc:1034] running on GCE node
I20260924 03:18:25.260818 28741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:25.262216 28741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:25.263428 28741 hybrid_clock.cc:648] HybridClock initialized: now 1790219905263385 us; error 47 us; skew 500 ppm
I20260924 03:18:25.263756 28741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:25.266762 28741 init.cc:377] Logged in from keytab as kudu/127.14.227.189@KRBTEST.COM (short username kudu)
I20260924 03:18:25.268095 28741 webserver.cc:460] Webserver started at http://127.14.227.189:46139/ using document root <none> and password file <none>
I20260924 03:18:25.268735 28741 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:25.268873 28741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:25.269152 28741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:25.272190 28741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/data/instance:
uuid: "f9e33836c7d44ac383d539a51f3e4edf"
format_stamp: "Formatted at 2026-09-24 03:18:25 on dist-test-slave-vd0p"
server_key: "97484a6324d517ae992767440ea79545"
server_key_iv: "6c5c08b330f5ff9a4499ff6820eb8d0a"
server_key_version: "encryptionkey@0"
I20260924 03:18:25.272955 28741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/wal/instance:
uuid: "f9e33836c7d44ac383d539a51f3e4edf"
format_stamp: "Formatted at 2026-09-24 03:18:25 on dist-test-slave-vd0p"
server_key: "97484a6324d517ae992767440ea79545"
server_key_iv: "6c5c08b330f5ff9a4499ff6820eb8d0a"
server_key_version: "encryptionkey@0"
I20260924 03:18:25.277788 28741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 03:18:25.280920 28756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:25.282222 28741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:18:25.282397 28741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/wal
uuid: "f9e33836c7d44ac383d539a51f3e4edf"
format_stamp: "Formatted at 2026-09-24 03:18:25 on dist-test-slave-vd0p"
server_key: "97484a6324d517ae992767440ea79545"
server_key_iv: "6c5c08b330f5ff9a4499ff6820eb8d0a"
server_key_version: "encryptionkey@0"
I20260924 03:18:25.282580 28741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:25.300187 28741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:25.303871 28741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:25.304134 28741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:25.321722 28741 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.189:35839
I20260924 03:18:25.321751 28807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.189:35839 every 8 connection(s)
I20260924 03:18:25.323029 28741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
I20260924 03:18:25.326510 28808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:25.333155 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 28741
I20260924 03:18:25.333328 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-1/wal/instance
I20260924 03:18:25.332803 28808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:18:25.333843 15246 external_mini_cluster.cc:1428] Setting key bd6260490eff3d84b30d4d6e248dbf6f
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.227.188@KRBTEST.COM; defaulting to no policy
I20260924 03:18:25.356292 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.336003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.14.227.190:33455)
0924 03:18:25.336239 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.336246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.337429 (+ 1183us) server_negotiation.cc:407] Connection header received
0924 03:18:25.340354 (+ 2925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.340359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.340427 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.340517 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.342413 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.343536 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.344992 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.345770 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.345920 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.349149 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.349157 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.349166 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.349210 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.353309 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.354028 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.354033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.354039 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.354190 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.354589 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.354594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.354597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.355012 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.355181 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.355899 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.356070 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:18:25.357276 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.336404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33455 (local address 127.0.0.1:51904)
0924 03:18:25.337228 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.337276 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:25.340047 (+ 2771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:25.340780 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:25.340794 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:25.341599 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:25.342227 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.342246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.343675 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.343679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.344848 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.344859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.346021 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.346027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.346203 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.346880 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:25.346901 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:25.348933 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:25.353447 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.353459 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.353471 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.353881 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.354293 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.354296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.354298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.354465 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.355130 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:25.355135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:25.355443 (+ 308us) client_negotiation.cc:769] Sending connection context
0924 03:18:25.355716 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 03:18:25.356090 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":310,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.188@KRBTEST.COM" created.
I20260924 03:18:25.359216 28808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:18:25.375227 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.360063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47602)
0924 03:18:25.360223 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.360246 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:25.360348 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:25.361330 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:25.361337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:25.361842 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:25.362082 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.362097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.363424 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.363433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.364935 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.364946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.366112 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.366116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.366249 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.366825 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:25.366840 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:25.368672 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:25.372839 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.372842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.372845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.373189 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.373577 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.373580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.373582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.373714 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.374452 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:25.374455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:25.374566 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 03:18:25.374672 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 03:18:25.374917 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 03:18:25.375649 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.360267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.14.227.189:35839)
0924 03:18:25.360918 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.360924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.360949 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:18:25.360990 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.360995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.361174 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.361298 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.362241 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.363276 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.365071 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.365985 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.366134 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.368881 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.368895 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.368912 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.368957 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.372717 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.373311 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.373322 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.373328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.373474 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.373820 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.373828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.373834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.374342 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.374556 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.374864 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.375157 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":372,"threads_started":1}
I20260924 03:18:25.377990 28808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:18:25.379168 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.378681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54372)
0924 03:18:25.378889 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.378999 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 03:18:25.379473 28760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
I20260924 03:18:25.380911 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.380582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54382)
0924 03:18:25.380708 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.380776 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 03:18:25.381204 28808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab.
Entry for principal kudu/127.14.227.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab.
I20260924 03:18:25.388753 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.188:43703
--webserver_interface=127.14.227.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab
--principal=kudu/127.14.227.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:33455,127.14.227.189:35839,127.14.227.188:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:18:25.430492 28808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 1
I20260924 03:18:25.431730 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.431154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54396)
0924 03:18:25.431398 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.431544 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 03:18:25.432747 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.432373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54402)
0924 03:18:25.432532 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.432614 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 03:18:25.432977 28808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
I20260924 03:18:25.501221 28808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 2
I20260924 03:18:25.502483 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.502040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54410)
0924 03:18:25.502241 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.502332 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 03:18:25.503557 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.503249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54416)
0924 03:18:25.503401 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.503463 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 03:18:25.503868 28808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
I20260924 03:18:25.554029 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } attempt: 5
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:18:25.566887 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.554829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.14.227.189:35839)
0924 03:18:25.555001 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.555008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.555091 (+ 83us) server_negotiation.cc:407] Connection header received
0924 03:18:25.555301 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.555310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.555382 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.555496 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.556417 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.557089 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.558329 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.558762 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.558893 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.561909 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.561914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.561917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.561988 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.564413 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.565074 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.565081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.565085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.565206 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.565554 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.565561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.565564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.565839 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.566016 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.566358 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.566568 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:18:25.566890 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.554764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47604)
0924 03:18:25.554982 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.555002 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:25.555180 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:25.555519 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:25.555523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:25.556034 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:25.556237 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.556247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.557220 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.557227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.558191 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.558203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.558909 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.558916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.559119 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.559742 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:25.559761 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:25.561728 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:25.564591 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.564597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.564601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.564963 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.565309 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.565315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.565319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.565454 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.565974 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:25.565980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:25.566132 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 03:18:25.566269 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 03:18:25.566536 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 03:18:25.568151 28734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:18:25.569341 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.568982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54426)
0924 03:18:25.569153 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.569225 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 03:18:25.570253 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.569901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54432)
0924 03:18:25.570089 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.570158 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260924 03:18:25.570469 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
I20260924 03:18:25.595096 28808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 3
I20260924 03:18:25.596197 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.595769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54446)
0924 03:18:25.595944 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.596050 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 03:18:25.597141 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.596759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54458)
0924 03:18:25.596974 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.597042 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260924 03:18:25.597333 28808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
I20260924 03:18:25.621750 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 1
I20260924 03:18:25.622931 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.622470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54474)
0924 03:18:25.622691 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.622785 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 03:18:25.623864 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.623550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54486)
0924 03:18:25.623696 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.623761 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 03:18:25.624053 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
I20260924 03:18:25.692324 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 2
I20260924 03:18:25.693382 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.692920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54488)
0924 03:18:25.693138 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.693215 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 03:18:25.694331 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.694013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54496)
0924 03:18:25.694140 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.694232 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 03:18:25.694579 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
W20260924 03:18:25.707944 28815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:25.708256 28815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:25.708433 28815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:25.715673 28815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:25.715772 28815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:25.715817 28815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:25.715874 28815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:25.715916 28815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:25.725512 28815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:33455,127.14.227.189:35839,127.14.227.188:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab
--principal=kudu/127.14.227.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.188:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.227.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:25.726833 28815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:25.728066 28815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:25.739343 28820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:25.739563 28821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:25.739881 28823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:25.740541 28808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 4
I20260924 03:18:25.741807 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.741177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54502)
0924 03:18:25.741445 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.741571 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 03:18:25.742834 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.742543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54514)
0924 03:18:25.742672 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.742738 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 03:18:25.743046 28808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
I20260924 03:18:25.743564 28815 server_base.cc:1034] running on GCE node
I20260924 03:18:25.744271 28815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:25.745594 28815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:25.746774 28815 hybrid_clock.cc:648] HybridClock initialized: now 1790219905746711 us; error 56 us; skew 500 ppm
I20260924 03:18:25.747118 28815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:25.750176 28815 init.cc:377] Logged in from keytab as kudu/127.14.227.188@KRBTEST.COM (short username kudu)
I20260924 03:18:25.751633 28815 webserver.cc:460] Webserver started at http://127.14.227.188:38687/ using document root <none> and password file <none>
I20260924 03:18:25.752272 28815 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:25.752368 28815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:25.752620 28815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:25.755380 28815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/data/instance:
uuid: "afd56d05dd2d45a48699d7c843453dba"
format_stamp: "Formatted at 2026-09-24 03:18:25 on dist-test-slave-vd0p"
server_key: "89409eeb31161aa85ac39cba5aceced4"
server_key_iv: "c6cff89d6277a4ac4c1bc2738356fc74"
server_key_version: "encryptionkey@0"
I20260924 03:18:25.756088 28815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/wal/instance:
uuid: "afd56d05dd2d45a48699d7c843453dba"
format_stamp: "Formatted at 2026-09-24 03:18:25 on dist-test-slave-vd0p"
server_key: "89409eeb31161aa85ac39cba5aceced4"
server_key_iv: "c6cff89d6277a4ac4c1bc2738356fc74"
server_key_version: "encryptionkey@0"
I20260924 03:18:25.760805 28815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:18:25.764007 28830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:25.765311 28815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:18:25.765479 28815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/wal
uuid: "afd56d05dd2d45a48699d7c843453dba"
format_stamp: "Formatted at 2026-09-24 03:18:25 on dist-test-slave-vd0p"
server_key: "89409eeb31161aa85ac39cba5aceced4"
server_key_iv: "c6cff89d6277a4ac4c1bc2738356fc74"
server_key_version: "encryptionkey@0"
I20260924 03:18:25.765666 28815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:25.794162 28815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:25.798128 28815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:25.798425 28815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:25.800782 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 3
I20260924 03:18:25.801787 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.801346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54518)
0924 03:18:25.801536 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.801618 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 03:18:25.802879 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.802402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54534)
0924 03:18:25.802641 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.802730 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260924 03:18:25.803138 28734 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:43703: Network error: Client connection negotiation failed: client connection to 127.14.227.188:43703: connect: Connection refused (error 111)
I20260924 03:18:25.816316 28815 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.188:43703
I20260924 03:18:25.816418 28881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.188:43703 every 8 connection(s)
I20260924 03:18:25.817576 28815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
I20260924 03:18:25.820559 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 28815
I20260924 03:18:25.820742 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790219707325830-15246-0/minicluster-data/master-2/wal/instance
I20260924 03:18:25.821307 15246 external_mini_cluster.cc:1428] Setting key a36ab4c11b3c308270e9b69070e4e4fe
I20260924 03:18:25.821592 28882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:25.831003 28882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:25.846303 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.823167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.14.227.188:43703)
0924 03:18:25.823915 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.823925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.823964 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:18:25.826123 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.826148 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.826531 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.826831 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.828688 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.829914 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.831947 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.832690 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.832829 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.837709 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.837717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.837730 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.837779 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.841683 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.842383 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.842391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.842404 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.842601 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.842966 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.842971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.842974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.843656 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.843900 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.844418 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.844853 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":362,"threads_started":1}
I20260924 03:18:25.850090 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.833742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.14.227.190:33455)
0924 03:18:25.833979 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.833988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.834716 (+ 728us) server_negotiation.cc:407] Connection header received
0924 03:18:25.834891 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.834894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.834949 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.835024 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.836441 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.837194 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.839073 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.839602 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.839722 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.843104 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.843110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.843115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.843150 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.846712 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.847536 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.847541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.847545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.847695 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.848118 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.848123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.848126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.848460 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.848597 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.849194 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.849466 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:18:25.851220 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.833755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:33455 (local address 127.0.0.1:51908)
0924 03:18:25.834576 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.834626 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:25.834778 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:25.835106 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:25.835115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:25.835907 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:25.836284 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.836309 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.837337 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.837342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.838931 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.838949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.839794 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.839801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.839998 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.840547 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:25.840574 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:25.842950 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:25.846869 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.846879 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.846886 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.847410 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.847842 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.847847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.847851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.847993 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.848667 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:25.848674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:25.848845 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 03:18:25.850060 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0924 03:18:25.850827 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":396,"threads_started":1}
I20260924 03:18:25.854103 28882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:18:25.867221 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.849106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.14.227.190:33455)
0924 03:18:25.850360 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.850368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.850390 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:18:25.850444 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.850451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.850535 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.850654 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.852743 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.854070 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.855860 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.856312 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.856436 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.861347 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.861355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.861362 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.861401 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.864594 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.865205 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.865213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.865217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.865370 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.865714 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.865721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.865727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.866094 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.866310 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.866668 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.866826 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":3765,"threads_started":1}
I20260924 03:18:25.868414 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.854844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:35839 (local address 127.0.0.1:47608)
0924 03:18:25.855140 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.855157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:25.855259 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:25.855601 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:25.855606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:25.856026 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:25.856188 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.856199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.857233 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.857239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.858871 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.858890 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.859568 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.859575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.859833 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.860401 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:25.860417 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:25.862430 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:25.865765 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.865768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.865771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.866240 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.866662 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.866666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.866669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.866840 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.867510 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:25.867513 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:25.867633 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 03:18:25.867761 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 03:18:25.868029 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260924 03:18:25.868470 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.854849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.14.227.189:35839)
0924 03:18:25.855021 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.855026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.855241 (+ 215us) server_negotiation.cc:407] Connection header received
0924 03:18:25.855371 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.855376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.855470 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.855572 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.856355 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.857101 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.859015 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.859414 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.859508 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.862607 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.862615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.862622 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.862666 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.865637 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.866357 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.866361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.866365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.866532 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.866964 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.866968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.866972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.867311 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.867453 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.867861 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.868015 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:18:25.869486 28882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:18:25.883948 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.870217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54550)
0924 03:18:25.870353 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.870370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:25.870456 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:25.870941 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:25.870944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:25.871252 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:25.871386 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.871393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.872559 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.872565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.873909 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.873924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.874866 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.874873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.875057 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.875560 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:25.875599 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:25.878003 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:25.881101 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.881108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.881112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.881611 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.882072 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.882090 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.882095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.882241 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.882973 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:25.882976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:25.883077 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 03:18:25.883235 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 03:18:25.883537 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 03:18:25.884147 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.870498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.14.227.188:43703)
0924 03:18:25.870674 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.870682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.870703 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:18:25.870749 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.870753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.870818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.870916 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.871512 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.872445 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.874046 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.874684 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.874785 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.878163 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.878167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.878170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.878196 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.880993 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.881730 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.881735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.881738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.881902 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.882434 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.882451 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.882455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.882807 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.882930 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.883328 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.883739 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":56}
I20260924 03:18:25.885689 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.869173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.14.227.189:35839)
0924 03:18:25.869340 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.869346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.869364 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:18:25.869541 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.869544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.869593 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.869691 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.871207 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.872191 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.873906 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.874558 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.874682 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.879414 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.879419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.879422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.879454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.882749 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.883570 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.883575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.883579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.883730 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.884181 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.884185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.884188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.884499 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.884709 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.885047 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.885202 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:18:25.889611 28882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba: Bootstrap starting.
I20260924 03:18:25.890488 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:18:25.892831 28882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:25.894044 28882 log.cc:824] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:25.898258 28882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba: No bootstrap required, opened a new log
I20260924 03:18:25.913264 28882 raft_consensus.cc:348] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:25.913790 28882 raft_consensus.cc:374] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:25.913921 28882 raft_consensus.cc:729] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd56d05dd2d45a48699d7c843453dba, State: Initialized, Role: FOLLOWER
I20260924 03:18:25.915483 28882 consensus_queue.cc:260] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:25.918418 28737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.895613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.14.227.190:33455)
0924 03:18:25.895844 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.895853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.896184 (+ 331us) server_negotiation.cc:407] Connection header received
0924 03:18:25.896514 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.896519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.896605 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.896688 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.899671 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.901111 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.904074 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.905007 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.905131 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.909596 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.909604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.909609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.909650 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.914399 (+ 4749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.915686 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.915692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.915695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.915995 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.916532 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.916539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.916544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.916919 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.917183 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.917785 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.918062 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:18:25.922025 28897 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:25.922688 28897 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:18:25.922864 28896 raft_consensus.cc:482] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:25.923168 28896 raft_consensus.cc:504] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:25.924607 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.895409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.14.227.189:35839)
0924 03:18:25.895594 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.895602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.907646 (+ 12044us) server_negotiation.cc:407] Connection header received
0924 03:18:25.908152 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.908160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.908245 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.908315 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.910401 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.911560 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.913813 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.914520 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.914600 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.917939 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.917968 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.917976 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.918017 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.921100 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.922130 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.922135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.922138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.922298 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.922835 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.922841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.922844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.923202 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.923358 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.923982 (+ 624us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.924207 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:18:25.926874 28882 sys_catalog.cc:564] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:25.927129 28896 leader_election.cc:288] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3c7f5dc253401c9ccb5758e651e57f (127.14.227.190:33455), f9e33836c7d44ac383d539a51f3e4edf (127.14.227.189:35839)
I20260924 03:18:25.935252 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.896859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.14.227.188:43703)
0924 03:18:25.897045 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.897052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.914520 (+ 17468us) server_negotiation.cc:407] Connection header received
0924 03:18:25.914686 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.914692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.914772 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.914871 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.917206 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.918662 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.922166 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.923080 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.923191 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.926700 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.926706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.926711 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.926745 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.931061 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.932154 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.932159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.932163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.932330 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.932901 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.932909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.932915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.933340 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.933664 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.934496 (+ 832us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.934798 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 03:18:25.947135 28908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:18:25.947271 28908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:18:25.944072 28815 master_runner.cc:186] Error getting master registration for 127.14.227.190:33455: OK, instance_id { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" instance_seqno: 1790219904840864 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:18:25.955287 28815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:18:25.980338 28734 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 4
Sep 24 03:18:25 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
I20260924 03:18:25.996723 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.980917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.14.227.188:43703)
0924 03:18:25.981092 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:25.981098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:25.981162 (+ 64us) server_negotiation.cc:407] Connection header received
0924 03:18:25.981369 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:25.981374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:25.981501 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:25.981583 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:25.982442 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.983632 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.985376 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.986158 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.986283 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.990313 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:25.990319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:25.990324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:25.990368 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:25.993908 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.994711 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.994720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.994724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.994896 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:25.995273 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:25.995280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:25.995285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:25.995624 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:25.995753 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:25.996125 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 03:18:25.996320 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:18:25.996714 28738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:25.980886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54564)
0924 03:18:25.981088 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:25.981107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:25.981227 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:25.981611 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:25.981616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:25.982115 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:25.982296 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.982306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.983756 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.983762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.985230 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:25.985245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:25.986545 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:25.986551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:25.986759 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:25.987433 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:25.987456 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:25.990152 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:25.994064 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.994069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.994085 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.994600 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.995030 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:25.995035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:25.995039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:25.995170 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:25.995738 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:25.995743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:25.995882 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 03:18:25.996030 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 03:18:25.996301 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 03:18:26.002919 28734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f: Bootstrap starting.
I20260924 03:18:26.007089 28734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:26.008364 28734 log.cc:824] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:26.011657 28734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f: No bootstrap required, opened a new log
I20260924 03:18:26.022676 28734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:26.023069 28734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:26.023180 28734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3c7f5dc253401c9ccb5758e651e57f, State: Initialized, Role: FOLLOWER
I20260924 03:18:26.023919 28734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:26.025486 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.025977 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:18:26.026865 28734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:26.026930 28709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afd56d05dd2d45a48699d7c843453dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" is_pre_election: true
I20260924 03:18:26.027495 28709 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd56d05dd2d45a48699d7c843453dba in term 0.
I20260924 03:18:26.028119 28833 leader_election.cc:302] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3c7f5dc253401c9ccb5758e651e57f, afd56d05dd2d45a48699d7c843453dba; no voters:
I20260924 03:18:26.028569 28896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:18:26.028728 28896 raft_consensus.cc:482] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:18:26.028821 28896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:26.032112 28896 raft_consensus.cc:504] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:26.033453 28709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afd56d05dd2d45a48699d7c843453dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3c7f5dc253401c9ccb5758e651e57f"
I20260924 03:18:26.033653 28709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:26.037050 28709 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd56d05dd2d45a48699d7c843453dba in term 1.
I20260924 03:18:26.037493 28833 leader_election.cc:302] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3c7f5dc253401c9ccb5758e651e57f, afd56d05dd2d45a48699d7c843453dba; no voters:
I20260924 03:18:26.037801 28897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:26.040323 28897 raft_consensus.cc:686] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [term 1 LEADER]: Becoming Leader. State: Replica: afd56d05dd2d45a48699d7c843453dba, State: Running, Role: LEADER
I20260924 03:18:26.040891 28897 consensus_queue.cc:237] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:26.042227 28808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } attempt: 5
I20260924 03:18:26.043429 28896 leader_election.cc:288] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [CANDIDATE]: Term 1 election: Requested vote from peers 4d3c7f5dc253401c9ccb5758e651e57f (127.14.227.190:33455), f9e33836c7d44ac383d539a51f3e4edf (127.14.227.189:35839)
W20260924 03:18:26.050429 28836 tablet.cc:2369] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 03:18:26 dist-test-slave-vd0p krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
I20260924 03:18:26.059172 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader afd56d05dd2d45a48699d7c843453dba. Latest consensus state: current_term: 1 leader_uuid: "afd56d05dd2d45a48699d7c843453dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.059652 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:26.070535 28919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:18:26.070832 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:26.042721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:43703 (local address 127.0.0.1:54574)
0924 03:18:26.044912 (+ 2191us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:26.044951 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:26.045091 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:26.045763 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:26.045771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:26.046523 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:26.046788 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:26.046803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:26.048982 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:26.048990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:26.050480 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:26.050497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:26.051956 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:26.051962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:26.052193 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:26.053914 (+ 1721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:26.053981 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:26.056740 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:26.064332 (+ 7592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:26.064340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:26.064347 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:26.064979 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:26.066084 (+ 1105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:26.066092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:26.066098 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:26.066334 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:26.069899 (+ 3565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:26.069904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:26.070145 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 03:18:26.070282 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 03:18:26.070494 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 03:18:26.072101 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:26.043314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.14.227.188:43703)
0924 03:18:26.044740 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:26.044750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:26.045214 (+ 464us) server_negotiation.cc:407] Connection header received
0924 03:18:26.045284 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:26.045291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:26.045392 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:26.045499 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:26.047048 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:26.048518 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:26.051003 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:26.051600 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:26.051677 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:26.056972 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:26.056980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:26.056987 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:26.057027 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:26.064078 (+ 7051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:26.065266 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:26.065274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:26.065318 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:26.065782 (+ 464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:26.066689 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:26.066698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:26.066704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:26.069692 (+ 2988us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:26.071184 (+ 1492us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:26.071393 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 03:18:26.071652 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262}
I20260924 03:18:26.073189 28919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:18:26.076303 28808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf: Bootstrap starting.
I20260924 03:18:26.079706 28808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf: Neither blocks nor log segments found. Creating new log.
W20260924 03:18:26.080488 28688 tablet.cc:2369] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:18:26.083578 28709 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [term 1 FOLLOWER]: Refusing update from remote peer afd56d05dd2d45a48699d7c843453dba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:26.085233 28914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:26.086588 28808 log.cc:824] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:26.106544 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader afd56d05dd2d45a48699d7c843453dba. Latest consensus state: current_term: 1 leader_uuid: "afd56d05dd2d45a48699d7c843453dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.106878 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:18:26.107090 28808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf: No bootstrap required, opened a new log
I20260924 03:18:26.109426 28923 mvcc.cc:205] Tried to move back new op lower bound from 7332740735304212480 to 7332740735148072960. Current Snapshot: MvccSnapshot[applied={T|T < 7332740735304212480}]
I20260924 03:18:26.110239 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afd56d05dd2d45a48699d7c843453dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.110518 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:26.114746 28919 catalog_manager.cc:1347] Generated new cluster ID: 298b177699234a9993438cdce4a0c831
I20260924 03:18:26.114859 28919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:18:26.126753 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afd56d05dd2d45a48699d7c843453dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.127063 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:18:26.134052 28929 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:18:26.143811 28919 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:18:26.146265 28919 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:18:26.137642 28667 master_runner.cc:186] Error getting master registration for 127.14.227.189:35839: OK, instance_id { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" instance_seqno: 1790219905316707 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:18:26.152033 28667 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:18:26.143191 28808 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:26.154268 28808 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:26.154413 28808 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e33836c7d44ac383d539a51f3e4edf, State: Initialized, Role: FOLLOWER
I20260924 03:18:26.155069 28808 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } }
I20260924 03:18:26.156103 28932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.156502 28932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:18:26.157524 28808 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:26.157573 28782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afd56d05dd2d45a48699d7c843453dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e33836c7d44ac383d539a51f3e4edf"
I20260924 03:18:26.157982 28781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:26.161634 28781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [term 1 FOLLOWER]: Refusing update from remote peer afd56d05dd2d45a48699d7c843453dba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:26.162477 28783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afd56d05dd2d45a48699d7c843453dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e33836c7d44ac383d539a51f3e4edf" is_pre_election: true
I20260924 03:18:26.162842 28783 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd56d05dd2d45a48699d7c843453dba in term 1.
I20260924 03:18:26.163846 28914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:26.174757 28919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba: Generated new TSK 0
I20260924 03:18:26.175803 28919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:18:26.187361 28932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [sys.catalog]: SysCatalogTable state changed. Reason: New leader afd56d05dd2d45a48699d7c843453dba. Latest consensus state: current_term: 1 leader_uuid: "afd56d05dd2d45a48699d7c843453dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.187675 28932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:18:26.188040 28937 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afd56d05dd2d45a48699d7c843453dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.188460 28937 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:18:26.189642 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afd56d05dd2d45a48699d7c843453dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3c7f5dc253401c9ccb5758e651e57f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33455 } } peers { permanent_uuid: "f9e33836c7d44ac383d539a51f3e4edf" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 35839 } } peers { permanent_uuid: "afd56d05dd2d45a48699d7c843453dba" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 43703 } } }
I20260924 03:18:26.189908 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd56d05dd2d45a48699d7c843453dba [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:26.209070 28948 catalog_manager.cc:1259] Loaded cluster ID: 298b177699234a9993438cdce4a0c831
I20260924 03:18:26.209170 28948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf: loading cluster ID for follower catalog manager: success
I20260924 03:18:26.212883 28948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf: acquiring CA information for follower catalog manager: success
I20260924 03:18:26.215279 28948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9e33836c7d44ac383d539a51f3e4edf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:18:27.139024 28929 catalog_manager.cc:1259] Loaded cluster ID: 298b177699234a9993438cdce4a0c831
I20260924 03:18:27.139156 28929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f: loading cluster ID for follower catalog manager: success
I20260924 03:18:27.142176 28929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f: acquiring CA information for follower catalog manager: success
I20260924 03:18:27.144423 28929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d3c7f5dc253401c9ccb5758e651e57f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:18:27.286096 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 28667
I20260924 03:18:27.287393 28729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:27.396293 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 28667
I20260924 03:18:27.414129 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 28741
I20260924 03:18:27.415266 28803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:27.520309 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 28741
I20260924 03:18:27.540288 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 28815
I20260924 03:18:27.541306 28877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:27.647154 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 28815
2026-09-24T03:18:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4536 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:18:27 dist-test-slave-vd0p krb5kdc[28959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:18:27 dist-test-slave-vd0p krb5kdc[28959](info): set up 2 sockets
Sep 24 03:18:27 dist-test-slave-vd0p krb5kdc[28959](info): commencing operation
krb5kdc: starting...
W20260924 03:18:28.798475 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:18:28 dist-test-slave-vd0p krb5kdc[28959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:18:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:18:29.015651 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38689
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:38689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:29.327811 28972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:29.328145 28972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:29.328297 28972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:29.336036 28972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:29.336174 28972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:29.336287 28972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:29.336375 28972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:29.336442 28972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:29.346062 28972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:38689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:29.347276 28972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:29.348536 28972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:29.360116 28977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:29.360183 28978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:29.360236 28980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:29.361023 28972 server_base.cc:1034] running on GCE node
I20260924 03:18:29.361799 28972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:29.363143 28972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:29.364362 28972 hybrid_clock.cc:648] HybridClock initialized: now 1790219909364326 us; error 42 us; skew 500 ppm
I20260924 03:18:29.364704 28972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:29 dist-test-slave-vd0p krb5kdc[28959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219909, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:29.367636 28972 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:18:29.369060 28972 webserver.cc:460] Webserver started at http://127.14.227.190:34781/ using document root <none> and password file <none>
I20260924 03:18:29.369674 28972 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:29.369812 28972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:29.370165 28972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:29.372972 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "6449101b3c11402c9c17f9a5be005109"
format_stamp: "Formatted at 2026-09-24 03:18:29 on dist-test-slave-vd0p"
server_key: "4673894f278d234d3f3ed8dda9e0c285"
server_key_iv: "5fd7da5e8304f52c55f2c57a9c9ee045"
server_key_version: "encryptionkey@0"
I20260924 03:18:29.373718 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "6449101b3c11402c9c17f9a5be005109"
format_stamp: "Formatted at 2026-09-24 03:18:29 on dist-test-slave-vd0p"
server_key: "4673894f278d234d3f3ed8dda9e0c285"
server_key_iv: "5fd7da5e8304f52c55f2c57a9c9ee045"
server_key_version: "encryptionkey@0"
I20260924 03:18:29.378566 28972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:18:29.381701 28987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:29.383046 28972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:18:29.383212 28972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "6449101b3c11402c9c17f9a5be005109"
format_stamp: "Formatted at 2026-09-24 03:18:29 on dist-test-slave-vd0p"
server_key: "4673894f278d234d3f3ed8dda9e0c285"
server_key_iv: "5fd7da5e8304f52c55f2c57a9c9ee045"
server_key_version: "encryptionkey@0"
I20260924 03:18:29.383391 28972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:29.416688 28972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:29.420729 28972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:29.421020 28972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:29.439235 28972 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:38689
I20260924 03:18:29.439260 29038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:38689 every 8 connection(s)
I20260924 03:18:29.440550 28972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:18:29.444049 29039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:29.445587 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 28972
I20260924 03:18:29.445858 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:18:29.446727 15246 external_mini_cluster.cc:1428] Setting key 6c59a3650da709671514f2f783cae8af
I20260924 03:18:29.456825 29039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109: Bootstrap starting.
Sep 24 03:18:29 dist-test-slave-vd0p krb5kdc[28959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:29.460093 29039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:29.461385 29039 log.cc:824] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:29.464538 29039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109: No bootstrap required, opened a new log
I20260924 03:18:29.469034 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:29.448536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.14.227.190:38689)
0924 03:18:29.449263 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:29.449274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:29.449313 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:18:29.450969 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:29.450993 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:29.451368 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:29.451756 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:29.453224 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:29.454575 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:29.455907 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:29.456554 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:29.456732 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:29.461229 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:29.461236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:29.461250 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:29.461285 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:29.464616 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:29.465315 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:29.465338 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:29.465344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:29.465500 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:29.465916 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:29.465923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:29.465933 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:29.466899 (+ 966us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:29.467204 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:29.467632 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 03:18:29.468048 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":336,"threads_started":1}
I20260924 03:18:29.472837 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:18:29.473166 29039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6449101b3c11402c9c17f9a5be005109" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38689 } }
I20260924 03:18:29.473495 29039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:29.473605 29039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6449101b3c11402c9c17f9a5be005109, State: Initialized, Role: FOLLOWER
I20260924 03:18:29.474198 29039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6449101b3c11402c9c17f9a5be005109" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38689 } }
I20260924 03:18:29.474429 29039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:18:29.474575 29039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:18:29.474745 29039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:29.477274 29039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6449101b3c11402c9c17f9a5be005109" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38689 } }
I20260924 03:18:29.477738 29039 leader_election.cc:302] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6449101b3c11402c9c17f9a5be005109; no voters:
I20260924 03:18:29.478350 29039 leader_election.cc:288] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:18:29.478510 29044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:29.479182 29044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [term 1 LEADER]: Becoming Leader. State: Replica: 6449101b3c11402c9c17f9a5be005109, State: Running, Role: LEADER
I20260924 03:18:29.479694 29044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6449101b3c11402c9c17f9a5be005109" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38689 } }
I20260924 03:18:29.480404 29039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:29.482436 29045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6449101b3c11402c9c17f9a5be005109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6449101b3c11402c9c17f9a5be005109" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38689 } } }
I20260924 03:18:29.482460 29046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6449101b3c11402c9c17f9a5be005109. Latest consensus state: current_term: 1 leader_uuid: "6449101b3c11402c9c17f9a5be005109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6449101b3c11402c9c17f9a5be005109" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38689 } } }
I20260924 03:18:29.482878 29045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:29.482930 29046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109 [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:29.484476 29049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:18:29.488962 29049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:18:29.496855 29049 catalog_manager.cc:1347] Generated new cluster ID: 20eca167bae34129a246db0ea3b5d540
I20260924 03:18:29.496961 29049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:18:29.518234 29049 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:18:29.519532 29049 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:18:29.525411 29049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6449101b3c11402c9c17f9a5be005109: Generated new TSK 0
I20260924 03:18:29.526229 29049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:18:29.584627 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 28972
I20260924 03:18:29.585703 29034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:29.679999 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 28972
2026-09-24T03:18:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2029 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:18:29 dist-test-slave-vd0p krb5kdc[29067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:18:29 dist-test-slave-vd0p krb5kdc[29067](info): set up 2 sockets
Sep 24 03:18:29 dist-test-slave-vd0p krb5kdc[29067](info): commencing operation
krb5kdc: starting...
W20260924 03:18:30.750463 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:18:30 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:18:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:18:30.964741 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37649
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:37649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:31.279609 29080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:31.279862 29080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:31.280011 29080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:31.287556 29080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:31.287648 29080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:31.287747 29080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:31.287853 29080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:31.287933 29080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:31.297295 29080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:37649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:31.298523 29080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:31.299759 29080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:31.311141 29086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:31.311228 29088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:31.311203 29085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:31.312105 29080 server_base.cc:1034] running on GCE node
I20260924 03:18:31.312820 29080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:31.314201 29080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:31.315485 29080 hybrid_clock.cc:648] HybridClock initialized: now 1790219911315393 us; error 58 us; skew 500 ppm
I20260924 03:18:31.315845 29080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:31 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:31.323709 29080 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:18:31.325186 29080 webserver.cc:460] Webserver started at http://127.14.227.190:33317/ using document root <none> and password file <none>
I20260924 03:18:31.325873 29080 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:31.326047 29080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:31.326344 29080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:31.329126 29080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "bc7febfd13fd49248500f83468275e5c"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "82f79e856335159ea3168bbecaf11cc4"
server_key_iv: "a884dcb92173d2922695c7748e8716fe"
server_key_version: "encryptionkey@0"
I20260924 03:18:31.329847 29080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "bc7febfd13fd49248500f83468275e5c"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "82f79e856335159ea3168bbecaf11cc4"
server_key_iv: "a884dcb92173d2922695c7748e8716fe"
server_key_version: "encryptionkey@0"
I20260924 03:18:31.334715 29080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:18:31.337764 29095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:31.339030 29080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 03:18:31.339208 29080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "bc7febfd13fd49248500f83468275e5c"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "82f79e856335159ea3168bbecaf11cc4"
server_key_iv: "a884dcb92173d2922695c7748e8716fe"
server_key_version: "encryptionkey@0"
I20260924 03:18:31.339402 29080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:31.354542 29080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:31.358242 29080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:31.358525 29080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:31.375437 29080 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:37649
I20260924 03:18:31.375459 29146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:37649 every 8 connection(s)
I20260924 03:18:31.376802 29080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:18:31.380493 29147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:31.383309 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 29080
I20260924 03:18:31.383520 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:18:31.384325 15246 external_mini_cluster.cc:1428] Setting key a8ddb4af491f3fb4893ca194e0db36ee
I20260924 03:18:31.395778 29147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Bootstrap starting.
I20260924 03:18:31.399310 29147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:31.400523 29147 log.cc:824] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:31.403456 29147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: No bootstrap required, opened a new log
Sep 24 03:18:31 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:31.413255 29147 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:31.413709 29147 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:31.413820 29147 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7febfd13fd49248500f83468275e5c, State: Initialized, Role: FOLLOWER
I20260924 03:18:31.414506 29147 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:31.414793 29147 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:18:31.414944 29147 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:18:31.415138 29147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:31.418322 29147 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:31.418900 29147 leader_election.cc:302] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7febfd13fd49248500f83468275e5c; no voters:
I20260924 03:18:31.419286 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:31.386483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.14.227.190:37649)
0924 03:18:31.392116 (+ 5633us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:31.392132 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:31.392186 (+ 54us) server_negotiation.cc:407] Connection header received
0924 03:18:31.394545 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:31.394576 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:31.395158 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:31.395601 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:31.397753 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:31.399850 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:31.401455 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:31.402332 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:31.402500 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:31.410127 (+ 7627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:31.410145 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:31.410163 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:31.410201 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:31.414305 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:31.415326 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:31.415336 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:31.415343 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:31.415542 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:31.415920 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:31.415925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:31.415928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:31.416660 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:31.416959 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:31.417470 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 03:18:31.417861 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5343,"thread_start_us":5074,"threads_started":1}
I20260924 03:18:31.419603 29147 leader_election.cc:288] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:18:31.420276 29153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:31.420979 29153 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 1 LEADER]: Becoming Leader. State: Replica: bc7febfd13fd49248500f83468275e5c, State: Running, Role: LEADER
I20260924 03:18:31.421510 29147 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:31.421734 29153 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:31.431188 29154 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc7febfd13fd49248500f83468275e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } } }
I20260924 03:18:31.432755 29154 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:31.431609 29152 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7febfd13fd49248500f83468275e5c. Latest consensus state: current_term: 1 leader_uuid: "bc7febfd13fd49248500f83468275e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } } }
I20260924 03:18:31.433130 29152 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: This master's current role is: LEADER
W20260924 03:18:31.436695 29166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:18:31.436825 29166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:18:31.436934 29168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:18:31.439000 29168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:18:31.447418 29168 catalog_manager.cc:1347] Generated new cluster ID: 942aeb49e73c4b86a21a4747be7857ef
I20260924 03:18:31.447517 29168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:18:31.484215 29168 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:18:31.486155 29168 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:18:31.498261 29168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Generated new TSK 0
I20260924 03:18:31.499804 29168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:18:31.574122 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37649
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:31.919401 29173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:31.919668 29173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:31.919811 29173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:31.927274 29173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:31.927399 29173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:31.927558 29173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:18:31.937284 29173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:31.938478 29173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:31.939668 29173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:31.952956 29178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:31.953020 29179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:31.953650 29181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:31.954109 29173 server_base.cc:1034] running on GCE node
I20260924 03:18:31.954663 29173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:31.955577 29173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:31.956799 29173 hybrid_clock.cc:648] HybridClock initialized: now 1790219911956753 us; error 33 us; skew 500 ppm
I20260924 03:18:31.957094 29173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:31 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:31.960062 29173 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:18:31.961413 29173 webserver.cc:460] Webserver started at http://127.14.227.129:44741/ using document root <none> and password file <none>
I20260924 03:18:31.962143 29173 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:31.962287 29173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:31.962584 29173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:31.965327 29173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "302188598ba44b5598121d6a96896a30"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "56c7fa95c9ed525d238f4e1b313b5a39"
server_key_iv: "6d36e5438bb6132bf9cd248fe4646458"
server_key_version: "encryptionkey@0"
I20260924 03:18:31.966146 29173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "302188598ba44b5598121d6a96896a30"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "56c7fa95c9ed525d238f4e1b313b5a39"
server_key_iv: "6d36e5438bb6132bf9cd248fe4646458"
server_key_version: "encryptionkey@0"
I20260924 03:18:31.970934 29173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 03:18:31.974130 29188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:31.975354 29173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:18:31.975519 29173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "302188598ba44b5598121d6a96896a30"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "56c7fa95c9ed525d238f4e1b313b5a39"
server_key_iv: "6d36e5438bb6132bf9cd248fe4646458"
server_key_version: "encryptionkey@0"
I20260924 03:18:31.975705 29173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:32.002462 29173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:32.006687 29173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:32.006984 29173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:32.007927 29173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:32.009132 29173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:32.009200 29173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:32.009295 29173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:32.009368 29173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:32.042454 29173 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:45839
I20260924 03:18:32.042606 29300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:45839 every 8 connection(s)
I20260924 03:18:32.043709 29173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:18:32.048110 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 29173
I20260924 03:18:32.048321 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:18:32.049122 15246 external_mini_cluster.cc:1428] Setting key 7cedd0bfe3c7787709a564311b117013
Sep 24 03:18:32 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219911, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:32.065714 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:32.046201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:40271 (local address 127.14.227.190:37649)
0924 03:18:32.046667 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:32.046673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:32.047464 (+ 791us) server_negotiation.cc:407] Connection header received
0924 03:18:32.050125 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:32.050131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:32.050215 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:32.050325 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:32.052438 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:32.053134 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:32.055537 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:32.055922 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:32.056006 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:32.060071 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:32.060076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:32.060079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:32.060109 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:32.062703 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:32.063412 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:32.063421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:32.063423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:32.063545 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:32.063962 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:32.063967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:32.063970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:32.064270 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:32.064453 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:32.065138 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 03:18:32.065362 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":243,"threads_started":1}
I20260924 03:18:32.066687 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:32.046624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:40271)
0924 03:18:32.047312 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:32.047349 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:32.049812 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:32.050644 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:32.050658 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:32.051449 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:32.052260 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:32.052285 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:32.053288 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:32.053296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:32.055389 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:32.055406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:32.056124 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:32.056132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:32.056479 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:32.057209 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:32.057243 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:32.059865 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:32.062852 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:32.062860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:32.062876 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:32.063289 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:32.063654 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:32.063658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:32.063662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:32.063846 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:32.064432 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:32.064440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:32.064755 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 03:18:32.065036 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 03:18:32.065540 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":257,"threads_started":1}
I20260924 03:18:32.068204 29301 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37649
I20260924 03:18:32.068558 29301 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:32.069463 29301 heartbeater.cc:511] Master 127.14.227.190:37649 requested a full tablet report, sending...
I20260924 03:18:32.071684 29112 ts_manager.cc:194] Registered new tserver with Master: 302188598ba44b5598121d6a96896a30 (127.14.227.129:45839)
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:18:32.073135 29112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:40271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:18:32.101926 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37649
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:32.420387 29307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:32.420707 29307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:32.420856 29307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:32.427898 29307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:32.428031 29307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:32.428189 29307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:18:32.437156 29307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:32.438344 29307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:32.439558 29307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:32.452015 29313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:32.451968 29315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:32.451941 29312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:32.452991 29307 server_base.cc:1034] running on GCE node
I20260924 03:18:32.453548 29307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:32.454538 29307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:32.455754 29307 hybrid_clock.cc:648] HybridClock initialized: now 1790219912455710 us; error 42 us; skew 500 ppm
I20260924 03:18:32.456037 29307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:32 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219912, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:32.458989 29307 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:18:32.460287 29307 webserver.cc:460] Webserver started at http://127.14.227.130:37271/ using document root <none> and password file <none>
I20260924 03:18:32.460892 29307 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:32.460988 29307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:32.461267 29307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:32.463995 29307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "cda96edc9be541fc92ee4390495a5390"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "d9de3c03390d91bb287110f696fca4fd"
server_key_iv: "4f546d18b16aac567e2b1cd7479ed01a"
server_key_version: "encryptionkey@0"
I20260924 03:18:32.464694 29307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "cda96edc9be541fc92ee4390495a5390"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "d9de3c03390d91bb287110f696fca4fd"
server_key_iv: "4f546d18b16aac567e2b1cd7479ed01a"
server_key_version: "encryptionkey@0"
I20260924 03:18:32.469266 29307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 03:18:32.472543 29322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:32.473810 29307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:18:32.473985 29307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "cda96edc9be541fc92ee4390495a5390"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "d9de3c03390d91bb287110f696fca4fd"
server_key_iv: "4f546d18b16aac567e2b1cd7479ed01a"
server_key_version: "encryptionkey@0"
I20260924 03:18:32.474164 29307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:32.496258 29307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:32.500134 29307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:32.500427 29307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:32.501364 29307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:32.502736 29307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:32.502813 29307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:32.502988 29307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:32.503037 29307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:32.536988 29307 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:39895
I20260924 03:18:32.537031 29434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:39895 every 8 connection(s)
I20260924 03:18:32.538389 29307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:18:32.542687 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 29307
I20260924 03:18:32.542898 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:18:32.543486 15246 external_mini_cluster.cc:1428] Setting key f3f416291327bb91025b3adcbcd68ed7
Sep 24 03:18:32 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219912, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:32.561511 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:32.540833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:43965 (local address 127.14.227.190:37649)
0924 03:18:32.541011 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:32.541016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:32.542170 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 03:18:32.544825 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:32.544831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:32.544920 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:32.545030 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:32.546962 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:32.547723 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:32.550090 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:32.550486 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:32.550578 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:32.554727 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:32.554735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:32.554742 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:32.554784 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:32.558404 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:32.559083 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:32.559091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:32.559098 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:32.559249 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:32.559646 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:32.559653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:32.559659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:32.560042 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:32.560217 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:32.560834 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 03:18:32.561076 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:18:32.562145 29437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:32.541324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:43965)
0924 03:18:32.542025 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:32.542064 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:32.544566 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:32.545255 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:32.545270 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:32.545935 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:32.546790 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:32.546815 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:32.547874 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:32.547882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:32.549913 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:32.549929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:32.550701 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:32.550710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:32.551017 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:32.551768 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:32.551801 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:32.554504 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:32.558549 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:32.558556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:32.558567 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:32.558957 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:32.559348 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:32.559351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:32.559353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:32.559509 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:32.560160 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:32.560165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:32.560448 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 03:18:32.560677 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 03:18:32.561029 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":264,"threads_started":1}
I20260924 03:18:32.563618 29435 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37649
I20260924 03:18:32.563907 29435 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:32.564702 29435 heartbeater.cc:511] Master 127.14.227.190:37649 requested a full tablet report, sending...
I20260924 03:18:32.566258 29112 ts_manager.cc:194] Registered new tserver with Master: cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
I20260924 03:18:32.567492 29112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:43965
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:18:32.596382 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37649
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:32.910806 29440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:32.911062 29440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:32.911155 29440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:32.918102 29440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:32.918192 29440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:32.918300 29440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:18:32.927736 29440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:32.928946 29440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:32.930260 29440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:32.942623 29446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:32.942580 29448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:32.942670 29445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:32.943634 29440 server_base.cc:1034] running on GCE node
I20260924 03:18:32.944319 29440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:32.945173 29440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:32.946357 29440 hybrid_clock.cc:648] HybridClock initialized: now 1790219912946322 us; error 28 us; skew 500 ppm
I20260924 03:18:32.946662 29440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:32 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219912, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:32.949640 29440 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:18:32.951176 29440 webserver.cc:460] Webserver started at http://127.14.227.131:37527/ using document root <none> and password file <none>
I20260924 03:18:32.951841 29440 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:32.951962 29440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:32.952250 29440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:32.955072 29440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "9027d7b00f5040d994f1c937e03c9009"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "ce04de251268cd1d5e07c80092825dfb"
server_key_iv: "5faaa4f0a6d331a675482c617982b8f6"
server_key_version: "encryptionkey@0"
I20260924 03:18:32.955816 29440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "9027d7b00f5040d994f1c937e03c9009"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "ce04de251268cd1d5e07c80092825dfb"
server_key_iv: "5faaa4f0a6d331a675482c617982b8f6"
server_key_version: "encryptionkey@0"
I20260924 03:18:32.960839 29440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:18:32.964223 29455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:32.965553 29440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:18:32.965698 29440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "9027d7b00f5040d994f1c937e03c9009"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "ce04de251268cd1d5e07c80092825dfb"
server_key_iv: "5faaa4f0a6d331a675482c617982b8f6"
server_key_version: "encryptionkey@0"
I20260924 03:18:32.965866 29440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:32.980665 29440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:32.984447 29440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:32.984723 29440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:32.985597 29440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:32.986935 29440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:32.987041 29440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:32.987180 29440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:32.987257 29440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:33.020223 29440 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:36541
I20260924 03:18:33.020179 29567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:36541 every 8 connection(s)
I20260924 03:18:33.022053 29440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:18:33.025569 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 29440
I20260924 03:18:33.025763 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:18:33.026412 15246 external_mini_cluster.cc:1428] Setting key e42ef40f3842e737742de22ab8a877d1
Sep 24 03:18:33 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219912, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:33.043741 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.025064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:54893 (local address 127.14.227.190:37649)
0924 03:18:33.025320 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.025327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.026721 (+ 1394us) server_negotiation.cc:407] Connection header received
0924 03:18:33.029211 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.029215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.029437 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.029577 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:33.031183 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.032000 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.033481 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.033903 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.034101 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.037288 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:33.037293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:33.037296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:33.037336 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:33.040040 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.041058 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.041065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.041069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.041223 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.041774 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.041779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.041782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.042257 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:33.042413 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.043088 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.043322 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:18:33.044507 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.025602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:54893)
0924 03:18:33.026509 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:33.026563 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:33.028921 (+ 2358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:33.029832 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:33.029841 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:33.030411 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:33.031009 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.031026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.032158 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.032162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.033343 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.033354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.034179 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.034184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.034381 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.035064 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:33.035093 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:33.037113 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:33.040228 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:33.040241 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:33.040258 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:33.040909 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:33.041344 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:33.041351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:33.041355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:33.041616 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:33.042375 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:33.042380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:33.042686 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 03:18:33.042966 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 03:18:33.043439 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":357,"threads_started":1}
I20260924 03:18:33.045924 29568 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37649
I20260924 03:18:33.046255 29568 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:33.046960 29568 heartbeater.cc:511] Master 127.14.227.190:37649 requested a full tablet report, sending...
I20260924 03:18:33.048466 29112 ts_manager.cc:194] Registered new tserver with Master: 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541)
I20260924 03:18:33.049266 29112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:54893
I20260924 03:18:33.054589 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:18:33.071370 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.059228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.14.227.190:37649)
0924 03:18:33.059378 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.059382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.059825 (+ 443us) server_negotiation.cc:407] Connection header received
0924 03:18:33.059955 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.059959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.060011 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.060089 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:33.061425 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.062207 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.063602 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.063995 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.064083 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.066376 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:33.066380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:33.066383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:33.066422 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:33.068889 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.069558 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.069562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.069566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.069729 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.070229 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.070233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.070235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.070531 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:33.070712 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.070982 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.071126 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 03:18:33.076270 29301 heartbeater.cc:503] Master 127.14.227.190:37649 was elected leader, sending a full tablet report...
I20260924 03:18:33.075073 29112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:18:33.082444 29112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:18:33.120743 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.111944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.14.227.130:39895)
0924 03:18:33.112744 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.112765 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.113491 (+ 726us) server_negotiation.cc:407] Connection header received
0924 03:18:33.113829 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.113839 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.114089 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.114289 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:33.114828 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.115861 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.118350 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.119540 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.119714 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.119880 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.120217 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.120391 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":503,"threads_started":1}
I20260924 03:18:33.121073 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.112727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:39895 (local address 127.0.0.1:38684)
0924 03:18:33.113382 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:33.113402 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:33.113623 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:33.114281 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:33.114285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:33.114308 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:33.114678 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.114687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.116006 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.116011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.118194 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.118206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.119758 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.119766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.119961 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.120017 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 03:18:33.120163 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 03:18:33.120361 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":64,"thread_start_us":353,"threads_started":1}
I20260924 03:18:33.122363 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.094262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.14.227.129:45839)
0924 03:18:33.112476 (+ 18214us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.112486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.112514 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:18:33.112608 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.112615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.112951 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.113158 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:33.114150 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.115852 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.119043 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.120369 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.120568 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.120733 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.121782 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.121979 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17970,"thread_start_us":17806,"threads_started":1}
I20260924 03:18:33.122359 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.094113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:36541 (local address 127.0.0.1:37566)
0924 03:18:33.111574 (+ 17461us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:33.111628 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:33.111857 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:33.113623 (+ 1766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:33.113629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:33.113652 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:33.114270 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.114285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.116334 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.116340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.118889 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.118899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.121133 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.121138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.121618 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.121652 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:18:33.121726 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:18:33.121825 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17300,"mutex_wait_us":400,"spinlock_wait_cycles":4352,"thread_start_us":15531,"threads_started":1}
I20260924 03:18:33.122777 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.094852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.14.227.131:36541)
0924 03:18:33.112654 (+ 17802us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.112664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.112698 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:18:33.112817 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.112828 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.113079 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.113283 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:33.114563 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.116140 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.119107 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.120361 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.120560 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.120740 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.122339 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.122510 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17595,"thread_start_us":17900,"threads_started":1}
I20260924 03:18:33.122977 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.093965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33900)
0924 03:18:33.111572 (+ 17607us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:33.111632 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:33.111860 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:33.113313 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:33.113320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:33.113356 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:33.113900 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.113914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.116042 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.116048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.118861 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.118883 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.121023 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.121055 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.121569 (+ 514us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.121605 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:18:33.121688 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:18:33.121817 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17224,"mutex_wait_us":190,"spinlock_wait_cycles":6400,"thread_start_us":17302,"threads_started":1}
I20260924 03:18:33.125402 29370 tablet_service.cc:1467] Processing CreateTablet for tablet d297dda02c83436eb7e2d31006d44828 (DEFAULT_TABLE table=test-table [id=118f3cd14c1a4b9abf93ff9432e2c708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:33.127019 29370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d297dda02c83436eb7e2d31006d44828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:33.126675 29236 tablet_service.cc:1467] Processing CreateTablet for tablet d297dda02c83436eb7e2d31006d44828 (DEFAULT_TABLE table=test-table [id=118f3cd14c1a4b9abf93ff9432e2c708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:33.128161 29236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d297dda02c83436eb7e2d31006d44828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:33.129056 29503 tablet_service.cc:1467] Processing CreateTablet for tablet d297dda02c83436eb7e2d31006d44828 (DEFAULT_TABLE table=test-table [id=118f3cd14c1a4b9abf93ff9432e2c708]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:33.130198 29503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d297dda02c83436eb7e2d31006d44828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:33.144613 29586 tablet_bootstrap.cc:492] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Bootstrap starting.
I20260924 03:18:33.147818 29588 tablet_bootstrap.cc:492] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: Bootstrap starting.
I20260924 03:18:33.148298 29586 tablet_bootstrap.cc:654] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:33.149562 29586 log.cc:824] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:33.158219 29587 tablet_bootstrap.cc:492] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: Bootstrap starting.
I20260924 03:18:33.159960 29588 tablet_bootstrap.cc:654] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:33.161016 29588 log.cc:824] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:33.161216 29586 tablet_bootstrap.cc:492] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: No bootstrap required, opened a new log
I20260924 03:18:33.161545 29586 ts_tablet_manager.cc:1397] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s
I20260924 03:18:33.161967 29587 tablet_bootstrap.cc:654] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:33.163264 29587 log.cc:824] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:33.163897 29588 tablet_bootstrap.cc:492] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: No bootstrap required, opened a new log
I20260924 03:18:33.164175 29588 ts_tablet_manager.cc:1397] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260924 03:18:33.166569 29587 tablet_bootstrap.cc:492] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: No bootstrap required, opened a new log
I20260924 03:18:33.166844 29587 ts_tablet_manager.cc:1397] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260924 03:18:33.171617 29588 raft_consensus.cc:348] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.172676 29588 raft_consensus.cc:374] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:33.172804 29588 raft_consensus.cc:729] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda96edc9be541fc92ee4390495a5390, State: Initialized, Role: FOLLOWER
I20260924 03:18:33.172979 29586 raft_consensus.cc:348] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.173435 29586 raft_consensus.cc:374] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:33.173566 29586 raft_consensus.cc:729] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9027d7b00f5040d994f1c937e03c9009, State: Initialized, Role: FOLLOWER
I20260924 03:18:33.173483 29588 consensus_queue.cc:260] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.174206 29586 consensus_queue.cc:260] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.175500 29568 heartbeater.cc:503] Master 127.14.227.190:37649 was elected leader, sending a full tablet report...
I20260924 03:18:33.176083 29586 ts_tablet_manager.cc:1428] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260924 03:18:33.177699 29587 raft_consensus.cc:348] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.178170 29587 raft_consensus.cc:374] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:33.178279 29587 raft_consensus.cc:729] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302188598ba44b5598121d6a96896a30, State: Initialized, Role: FOLLOWER
I20260924 03:18:33.178898 29587 consensus_queue.cc:260] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.180764 29435 heartbeater.cc:503] Master 127.14.227.190:37649 was elected leader, sending a full tablet report...
I20260924 03:18:33.181633 29588 ts_tablet_manager.cc:1428] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260924 03:18:33.183220 29587 ts_tablet_manager.cc:1428] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260924 03:18:33.200825 29592 raft_consensus.cc:482] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:33.201140 29592 raft_consensus.cc:504] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.202808 29592 leader_election.cc:288] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 302188598ba44b5598121d6a96896a30 (127.14.227.129:45839), 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541)
I20260924 03:18:33.211480 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.203286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:44687 (local address 127.14.227.131:36541)
0924 03:18:33.203478 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.203485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.204378 (+ 893us) server_negotiation.cc:407] Connection header received
0924 03:18:33.204508 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.204513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.204580 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.204659 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:33.205208 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.206332 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.209083 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.210350 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.210514 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.210585 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.211097 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.211228 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:18:33.211483 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.203285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:37679 (local address 127.14.227.129:45839)
0924 03:18:33.203478 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.203485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.204328 (+ 843us) server_negotiation.cc:407] Connection header received
0924 03:18:33.204811 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.204814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.204872 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.204938 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:33.205545 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.206570 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.209079 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.210363 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.210525 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.210587 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.211097 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.211228 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:18:33.211870 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.203280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.14.227.130:37679)
0924 03:18:33.204203 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:33.204220 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:33.204382 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:33.204987 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:33.204992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:33.205021 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:33.205389 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.205396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.206690 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.206696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.208905 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.208918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.210626 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.210632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.210881 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.210953 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 03:18:33.211049 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:18:33.211256 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":237,"thread_start_us":515,"threads_started":1}
I20260924 03:18:33.211891 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.203265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:36541 (local address 127.14.227.130:44687)
0924 03:18:33.204199 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:33.204220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:33.204387 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:33.204690 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:33.204694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:33.204713 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:33.205060 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.205073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.206446 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.206450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.208904 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:33.208918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.210638 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.210644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:33.210874 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.210957 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 03:18:33.211049 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 03:18:33.211256 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":180,"spinlock_wait_cycles":36736,"thread_start_us":414,"threads_started":1}
I20260924 03:18:33.212858 29523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d297dda02c83436eb7e2d31006d44828" candidate_uuid: "cda96edc9be541fc92ee4390495a5390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9027d7b00f5040d994f1c937e03c9009" is_pre_election: true
I20260924 03:18:33.212862 29256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d297dda02c83436eb7e2d31006d44828" candidate_uuid: "cda96edc9be541fc92ee4390495a5390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302188598ba44b5598121d6a96896a30" is_pre_election: true
I20260924 03:18:33.213342 29523 raft_consensus.cc:2454] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda96edc9be541fc92ee4390495a5390 in term 0.
I20260924 03:18:33.213342 29256 raft_consensus.cc:2454] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cda96edc9be541fc92ee4390495a5390 in term 0.
I20260924 03:18:33.214058 29323 leader_election.cc:302] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 302188598ba44b5598121d6a96896a30, 9027d7b00f5040d994f1c937e03c9009, cda96edc9be541fc92ee4390495a5390; no voters:
I20260924 03:18:33.214468 29592 raft_consensus.cc:2781] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:18:33.214618 29592 raft_consensus.cc:482] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:18:33.214737 29592 raft_consensus.cc:3037] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:33.217301 29592 raft_consensus.cc:504] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.217978 29592 leader_election.cc:288] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [CANDIDATE]: Term 1 election: Requested vote from peers 302188598ba44b5598121d6a96896a30 (127.14.227.129:45839), 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541)
I20260924 03:18:33.218317 29256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d297dda02c83436eb7e2d31006d44828" candidate_uuid: "cda96edc9be541fc92ee4390495a5390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302188598ba44b5598121d6a96896a30"
I20260924 03:18:33.218437 29523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d297dda02c83436eb7e2d31006d44828" candidate_uuid: "cda96edc9be541fc92ee4390495a5390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9027d7b00f5040d994f1c937e03c9009"
I20260924 03:18:33.218526 29256 raft_consensus.cc:3037] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:33.218663 29523 raft_consensus.cc:3037] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:33.220683 29256 raft_consensus.cc:2454] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda96edc9be541fc92ee4390495a5390 in term 1.
I20260924 03:18:33.220791 29523 raft_consensus.cc:2454] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cda96edc9be541fc92ee4390495a5390 in term 1.
I20260924 03:18:33.221185 29325 leader_election.cc:302] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302188598ba44b5598121d6a96896a30, cda96edc9be541fc92ee4390495a5390; no voters:
I20260924 03:18:33.221542 29592 raft_consensus.cc:2781] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:33.222054 29592 raft_consensus.cc:686] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 1 LEADER]: Becoming Leader. State: Replica: cda96edc9be541fc92ee4390495a5390, State: Running, Role: LEADER
I20260924 03:18:33.222493 29592 consensus_queue.cc:237] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:33.226862 29111 catalog_manager.cc:5495] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 reported cstate change: term changed from 0 to 1, leader changed from <none> to cda96edc9be541fc92ee4390495a5390 (127.14.227.130). New cstate: current_term: 1 leader_uuid: "cda96edc9be541fc92ee4390495a5390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } health_report { overall_health: HEALTHY } } }
I20260924 03:18:33.258060 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.252743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.14.227.130:39895)
0924 03:18:33.252950 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.252958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.252991 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:18:33.253095 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.253099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.253170 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.253324 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:18:33.253713 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.254471 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.255889 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.256308 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.256448 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.256943 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:18:33.257438 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:18:33.257600 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.257655 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.257733 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:18:33.265771 29523 raft_consensus.cc:1261] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 1 FOLLOWER]: Refusing update from remote peer cda96edc9be541fc92ee4390495a5390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:33.265806 29256 raft_consensus.cc:1261] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Refusing update from remote peer cda96edc9be541fc92ee4390495a5390: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:33.266736 29597 consensus_queue.cc:1035] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:33.267066 29592 consensus_queue.cc:1035] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [LEADER]: Connected to new peer: Peer: permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:18:33.274173 29569 tablet.cc:2369] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:18:33.277778 29600 mvcc.cc:205] Tried to move back new op lower bound from 7332740764727496704 to 7332740764563382272. Current Snapshot: MvccSnapshot[applied={T|T < 7332740764727496704}]
I20260924 03:18:33.291042 29111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:18:33.291326 29111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:18:33.294663 29111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Sending DeleteTablet for 3 replicas of tablet d297dda02c83436eb7e2d31006d44828
I20260924 03:18:33.295725 29503 tablet_service.cc:1514] Processing DeleteTablet for tablet d297dda02c83436eb7e2d31006d44828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:33 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:37566
I20260924 03:18:33.295917 29370 tablet_service.cc:1514] Processing DeleteTablet for tablet d297dda02c83436eb7e2d31006d44828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:33 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38684
I20260924 03:18:33.295948 29236 tablet_service.cc:1514] Processing DeleteTablet for tablet d297dda02c83436eb7e2d31006d44828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:33 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:33900
I20260924 03:18:33.296241 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 29173
I20260924 03:18:33.297320 29609 tablet_replica.cc:330] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: stopping tablet replica
I20260924 03:18:33.297823 29609 raft_consensus.cc:2229] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:18:33.298408 29609 raft_consensus.cc:2258] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:33.299207 29610 tablet_replica.cc:330] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: stopping tablet replica
I20260924 03:18:33.299590 29610 raft_consensus.cc:2229] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:18:33.300253 29610 raft_consensus.cc:2258] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:33.300946 29611 tablet_replica.cc:330] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: stopping tablet replica
I20260924 03:18:33.301312 29611 raft_consensus.cc:2229] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:18:33.302074 29611 raft_consensus.cc:2258] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:33.303802 29609 ts_tablet_manager.cc:1905] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:33.303850 29610 ts_tablet_manager.cc:1905] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:33.310050 29611 ts_tablet_manager.cc:1905] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:33.310752 29609 ts_tablet_manager.cc:1918] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:33.310906 29609 log.cc:1196] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/d297dda02c83436eb7e2d31006d44828
I20260924 03:18:33.311349 29610 ts_tablet_manager.cc:1918] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:33.311446 29609 ts_tablet_manager.cc:1939] T d297dda02c83436eb7e2d31006d44828 P 9027d7b00f5040d994f1c937e03c9009: Deleting consensus metadata
I20260924 03:18:33.311506 29610 log.cc:1196] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/d297dda02c83436eb7e2d31006d44828
I20260924 03:18:33.312041 29610 ts_tablet_manager.cc:1939] T d297dda02c83436eb7e2d31006d44828 P cda96edc9be541fc92ee4390495a5390: Deleting consensus metadata
I20260924 03:18:33.313241 29096 catalog_manager.cc:4841] TS 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541): tablet d297dda02c83436eb7e2d31006d44828 (table test-table [id=118f3cd14c1a4b9abf93ff9432e2c708]) successfully deleted
I20260924 03:18:33.313520 29098 catalog_manager.cc:4841] TS cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895): tablet d297dda02c83436eb7e2d31006d44828 (table test-table [id=118f3cd14c1a4b9abf93ff9432e2c708]) successfully deleted
I20260924 03:18:33.316171 29611 ts_tablet_manager.cc:1918] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:33.316388 29611 log.cc:1196] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/d297dda02c83436eb7e2d31006d44828
I20260924 03:18:33.316823 29611 ts_tablet_manager.cc:1939] T d297dda02c83436eb7e2d31006d44828 P 302188598ba44b5598121d6a96896a30: Deleting consensus metadata
Sep 24 03:18:33 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:18:33.318174 29098 catalog_manager.cc:4841] TS 302188598ba44b5598121d6a96896a30 (127.14.227.129:45839): tablet d297dda02c83436eb7e2d31006d44828 (table test-table [id=118f3cd14c1a4b9abf93ff9432e2c708]) successfully deleted
I20260924 03:18:33.324016 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.298266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.14.227.129:45839)
0924 03:18:33.298491 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.298498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.301266 (+ 2768us) server_negotiation.cc:407] Connection header received
0924 03:18:33.301473 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.301478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.301570 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.301667 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:33.304274 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.305362 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.307326 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.307963 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.308092 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.318239 (+ 10147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:33.318251 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:33.318270 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:33.318321 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:33.321136 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.322074 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.322081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.322083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.322234 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.322623 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.322626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.322628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.323036 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:33.323238 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.323571 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.323739 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:18:33.324717 29296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:33.424841 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 29173
I20260924 03:18:33.442533 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 29307
Sep 24 03:18:33 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:18:33.460850 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.443512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.14.227.130:39895)
0924 03:18:33.443722 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.443728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.443748 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:18:33.443904 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.443909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.443991 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.444105 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:33.445392 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.446458 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.447712 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.448203 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.448339 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.454428 (+ 6089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:33.454439 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:33.454453 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:33.454487 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:33.457895 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.458892 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.458898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.458900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.459023 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.459440 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.459442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.459445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.459766 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:33.459985 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.460397 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.460557 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:18:33.461830 29430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:33.575577 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 29307
I20260924 03:18:33.594017 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 29440
Sep 24 03:18:33 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:18:33.610137 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:33.594781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.14.227.131:36541)
0924 03:18:33.594967 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:33.594976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:33.595001 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:18:33.595266 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:33.595278 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:33.595340 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:33.595451 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:33.596796 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.597513 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.598982 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:33.599380 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:33.599513 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:33.604290 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:33.604305 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:33.604321 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:33.604360 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:33.607445 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.608221 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.608228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.608230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.608377 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:33.608787 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:33.608790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:33.608792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:33.609127 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:33.609366 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:33.609643 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 03:18:33.609818 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:18:33.610812 29563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:33.698184 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 29440
I20260924 03:18:33.714604 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 29080
I20260924 03:18:33.715791 29142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:33.828444 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 29080
I20260924 03:18:33.849257 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37649
--webserver_interface=127.14.227.190
--webserver_port=33317
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:37649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:34.169070 29617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:34.169413 29617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:34.169569 29617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:34.176856 29617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:34.177013 29617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:34.177122 29617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:34.177223 29617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:34.177296 29617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:34.187323 29617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:37649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=33317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:34.188652 29617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:34.190099 29617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:34.212404 29626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:34.212319 29623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:34.212353 29624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:34.213155 29617 server_base.cc:1034] running on GCE node
I20260924 03:18:34.214005 29617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:34.215538 29617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:34.216737 29617 hybrid_clock.cc:648] HybridClock initialized: now 1790219914216679 us; error 50 us; skew 500 ppm
I20260924 03:18:34.217130 29617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:34 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219914, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:34.220569 29617 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:18:34.222169 29617 webserver.cc:460] Webserver started at http://127.14.227.190:33317/ using document root <none> and password file <none>
I20260924 03:18:34.222872 29617 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:34.222977 29617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:34.228680 29617 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 03:18:34.231642 29633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:34.232955 29617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 03:18:34.233139 29617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "bc7febfd13fd49248500f83468275e5c"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "82f79e856335159ea3168bbecaf11cc4"
server_key_iv: "a884dcb92173d2922695c7748e8716fe"
server_key_version: "encryptionkey@0"
I20260924 03:18:34.234262 29617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:34.254120 29617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:34.258916 29617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:34.259218 29617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:34.277380 29617 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:37649
I20260924 03:18:34.277400 29684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:37649 every 8 connection(s)
I20260924 03:18:34.278882 29617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:18:34.281308 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 29617
I20260924 03:18:34.282203 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:45839
--local_ip_for_outbound_sockets=127.14.227.129
--tserver_master_addrs=127.14.227.190:37649
--webserver_port=44741
--webserver_interface=127.14.227.129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:18:34.285841 29685 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:18:34.288903 29685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Bootstrap starting.
I20260924 03:18:34.307438 29685 log.cc:824] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:34.318907 29685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:18:34.319397 29685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Bootstrap complete.
I20260924 03:18:34.328469 29685 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:34.329282 29685 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7febfd13fd49248500f83468275e5c, State: Initialized, Role: FOLLOWER
I20260924 03:18:34.329855 29685 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:34.330158 29685 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:18:34.330283 29685 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:18:34.330473 29685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:18:34.334074 29685 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:34.334538 29685 leader_election.cc:302] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7febfd13fd49248500f83468275e5c; no voters:
I20260924 03:18:34.335179 29685 leader_election.cc:288] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:18:34.335305 29689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:18:34.336506 29689 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 2 LEADER]: Becoming Leader. State: Replica: bc7febfd13fd49248500f83468275e5c, State: Running, Role: LEADER
I20260924 03:18:34.337072 29689 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:34.337321 29685 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:34.339352 29691 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7febfd13fd49248500f83468275e5c. Latest consensus state: current_term: 2 leader_uuid: "bc7febfd13fd49248500f83468275e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } } }
I20260924 03:18:34.340080 29691 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:34.339781 29690 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bc7febfd13fd49248500f83468275e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } } }
I20260924 03:18:34.340715 29690 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:34.346246 29695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:18:34.355463 29695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:18:34.357532 29695 catalog_manager.cc:1259] Loaded cluster ID: 942aeb49e73c4b86a21a4747be7857ef
I20260924 03:18:34.357681 29695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:18:34.362428 29695 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 03:18:34.387396 29695 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 03:18:34.398231 29695 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:18:34.400318 29695 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Loaded TSK: 0
I20260924 03:18:34.402240 29695 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:18:34.637290 29687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:34.637655 29687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:34.637853 29687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:34.645430 29687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:34.645593 29687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:34.645821 29687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:18:34.655560 29687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:45839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=44741
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:34.657027 29687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:34.658273 29687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:34.672590 29712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:34.672495 29713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:34.672909 29715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:34.673836 29687 server_base.cc:1034] running on GCE node
I20260924 03:18:34.674423 29687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:34.675359 29687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:34.676573 29687 hybrid_clock.cc:648] HybridClock initialized: now 1790219914676492 us; error 62 us; skew 500 ppm
I20260924 03:18:34.676858 29687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:34 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219914, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:34.680037 29687 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:18:34.681581 29687 webserver.cc:460] Webserver started at http://127.14.227.129:44741/ using document root <none> and password file <none>
I20260924 03:18:34.682297 29687 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:34.682473 29687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:34.687585 29687 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 03:18:34.690150 29722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:34.691515 29687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:18:34.691669 29687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "302188598ba44b5598121d6a96896a30"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "56c7fa95c9ed525d238f4e1b313b5a39"
server_key_iv: "6d36e5438bb6132bf9cd248fe4646458"
server_key_version: "encryptionkey@0"
I20260924 03:18:34.692611 29687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:34.710600 29687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:34.714917 29687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:34.715268 29687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:34.716384 29687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:34.717783 29687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:34.717891 29687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:34.718040 29687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:34.718102 29687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:34.754515 29687 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:45839
I20260924 03:18:34.754594 29834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:45839 every 8 connection(s)
I20260924 03:18:34.756048 29687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:18:34.764065 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 29687
I20260924 03:18:34.764735 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:39895
--local_ip_for_outbound_sockets=127.14.227.130
--tserver_master_addrs=127.14.227.190:37649
--webserver_port=37271
--webserver_interface=127.14.227.130
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:18:34 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219914, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:34.780517 29838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:34.759376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:51553)
0924 03:18:34.760261 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:34.760306 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:34.762054 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:34.763442 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:34.763450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:34.763954 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:34.764557 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:34.764575 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:34.767065 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:34.767069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:34.768269 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:34.768283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:34.769789 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:34.769794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:34.770039 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:34.770719 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:34.770740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:34.772925 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:34.776238 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:34.776246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:34.776259 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:34.776737 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:34.777145 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:34.777152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:34.777155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:34.777341 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:34.778093 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:34.778101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:34.778423 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 03:18:34.778704 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 03:18:34.779221 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":382,"threads_started":1}
I20260924 03:18:34.780517 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:34.759082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:51553 (local address 127.14.227.190:37649)
0924 03:18:34.759727 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:34.759745 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:34.760433 (+ 688us) server_negotiation.cc:407] Connection header received
0924 03:18:34.762666 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:34.762679 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:34.763019 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:34.763300 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:34.764805 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:34.766906 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:34.768418 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:34.769580 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:34.769787 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:34.773187 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:34.773194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:34.773207 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:34.773240 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:34.776099 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:34.776872 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:34.776884 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:34.776889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:34.777034 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:34.777460 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:34.777468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:34.777470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:34.777896 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:34.778153 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:34.778894 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 03:18:34.779266 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":244,"threads_started":1}
I20260924 03:18:34.783792 29835 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37649
I20260924 03:18:34.784130 29835 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:34.785113 29835 heartbeater.cc:511] Master 127.14.227.190:37649 requested a full tablet report, sending...
I20260924 03:18:34.787532 29650 ts_manager.cc:194] Registered new tserver with Master: 302188598ba44b5598121d6a96896a30 (127.14.227.129:45839)
I20260924 03:18:34.788992 29650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:51553
W20260924 03:18:35.086011 29839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:35.086267 29839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:35.086409 29839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:35.093133 29839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:35.093227 29839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:35.093396 29839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:18:35.102849 29839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:39895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=37271
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:35.104022 29839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:35.105405 29839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:35.118309 29845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:35.118368 29844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:35.118363 29847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:35.119148 29839 server_base.cc:1034] running on GCE node
I20260924 03:18:35.119766 29839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:35.120678 29839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:35.121886 29839 hybrid_clock.cc:648] HybridClock initialized: now 1790219915121836 us; error 32 us; skew 500 ppm
I20260924 03:18:35.122265 29839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:35 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219915, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:35.125406 29839 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:18:35.126924 29839 webserver.cc:460] Webserver started at http://127.14.227.130:37271/ using document root <none> and password file <none>
I20260924 03:18:35.127627 29839 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:35.127784 29839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:35.133055 29839 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:18:35.135759 29854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:35.137154 29839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:18:35.137332 29839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "cda96edc9be541fc92ee4390495a5390"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "d9de3c03390d91bb287110f696fca4fd"
server_key_iv: "4f546d18b16aac567e2b1cd7479ed01a"
server_key_version: "encryptionkey@0"
I20260924 03:18:35.138513 29839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:35.155084 29839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:35.159087 29839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:35.159333 29839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:35.160264 29839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:35.161536 29839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:35.161655 29839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:35.161779 29839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:35.161877 29839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:35.194561 29839 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:39895
I20260924 03:18:35.194646 29966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:39895 every 8 connection(s)
I20260924 03:18:35.196004 29839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:18:35.197536 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 29839
I20260924 03:18:35.198207 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:36541
--local_ip_for_outbound_sockets=127.14.227.131
--tserver_master_addrs=127.14.227.190:37649
--webserver_port=37527
--webserver_interface=127.14.227.131
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:18:35 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219915, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:35.221477 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.198854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:60637 (local address 127.14.227.190:37649)
0924 03:18:35.199088 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.199097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.204370 (+ 5273us) server_negotiation.cc:407] Connection header received
0924 03:18:35.206259 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.206264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.206349 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.206464 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:35.208054 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.208793 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.210387 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.210798 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.210965 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.214501 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:35.214507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:35.214511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:35.214553 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:35.218237 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.218911 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.218919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.218925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.219090 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.219493 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.219501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.219504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.219890 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:35.220059 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.220730 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.221020 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:18:35.222131 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.199320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:60637)
0924 03:18:35.204186 (+ 4866us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:35.204229 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:35.205946 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:35.206695 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:35.206703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:35.207220 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:35.207862 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.207890 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.208955 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.208959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.210230 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.210248 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.211048 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.211054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.211286 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.211981 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:35.212012 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:35.214299 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:35.218388 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:35.218394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:35.218405 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:35.218763 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:35.219196 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:35.219199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:35.219201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:35.219372 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:35.220045 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:35.220050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:35.220338 (+ 288us) client_negotiation.cc:769] Sending connection context
0924 03:18:35.220569 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 03:18:35.220953 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4637,"thread_start_us":4492,"threads_started":1}
I20260924 03:18:35.223503 29967 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37649
I20260924 03:18:35.223809 29967 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:35.224646 29967 heartbeater.cc:511] Master 127.14.227.190:37649 requested a full tablet report, sending...
I20260924 03:18:35.226182 29650 ts_manager.cc:194] Registered new tserver with Master: cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
I20260924 03:18:35.226948 29650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:60637
W20260924 03:18:35.514868 29970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:35.515120 29970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:35.515295 29970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:35.522208 29970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:35.522320 29970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:35.522493 29970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:18:35.531642 29970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:36541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=37527
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:35.532836 29970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:35.534090 29970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:35.547003 29975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:35.547188 29976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:35.547061 29978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:35.547900 29970 server_base.cc:1034] running on GCE node
I20260924 03:18:35.548507 29970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:35.549291 29970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:35.550460 29970 hybrid_clock.cc:648] HybridClock initialized: now 1790219915550397 us; error 34 us; skew 500 ppm
I20260924 03:18:35.550787 29970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:35 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219915, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:35.553823 29970 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:18:35.555212 29970 webserver.cc:460] Webserver started at http://127.14.227.131:37527/ using document root <none> and password file <none>
I20260924 03:18:35.555763 29970 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:35.555910 29970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:35.561177 29970 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 03:18:35.563656 29985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:35.564929 29970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:18:35.565092 29970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "9027d7b00f5040d994f1c937e03c9009"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "ce04de251268cd1d5e07c80092825dfb"
server_key_iv: "5faaa4f0a6d331a675482c617982b8f6"
server_key_version: "encryptionkey@0"
I20260924 03:18:35.566015 29970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:35.591078 29970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:35.595095 29970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:35.595367 29970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:35.596489 29970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:35.597957 29970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:35.598037 29970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:35.598183 29970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:35.598270 29970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:35.630195 29970 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:36541
I20260924 03:18:35.630280 30097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:36541 every 8 connection(s)
I20260924 03:18:35.631631 29970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:18:35.640040 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 29970
Sep 24 03:18:35 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219915, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:35.652758 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.634048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:38373 (local address 127.14.227.190:37649)
0924 03:18:35.634257 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.634263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.635402 (+ 1139us) server_negotiation.cc:407] Connection header received
0924 03:18:35.637218 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.637227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.637324 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.637428 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:35.639061 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.639837 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.641943 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.642536 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.642626 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.646591 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:35.646597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:35.646602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:35.646640 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:35.649292 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.650251 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.650254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.650256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.650370 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.650824 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.650829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.650839 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.651165 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:35.651314 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.652245 (+ 931us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.652528 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:18:35.654103 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.634588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:38373)
0924 03:18:35.635254 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:35.635299 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:35.636844 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:35.637645 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:35.637653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:35.638282 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:35.638883 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.638900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.640023 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.640029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.641795 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.641810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.642760 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.642768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.643115 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.643870 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:35.643903 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:35.646357 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:35.649450 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:35.649462 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:35.649479 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:35.650133 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:35.650527 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:35.650534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:35.650538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:35.650706 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:35.651361 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:35.651372 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:35.651773 (+ 401us) client_negotiation.cc:769] Sending connection context
0924 03:18:35.652096 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 03:18:35.652684 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":258,"threads_started":1}
I20260924 03:18:35.655483 30098 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37649
I20260924 03:18:35.655802 30098 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:35.656562 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.640948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.14.227.190:37649)
0924 03:18:35.641914 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.641919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.641932 (+ 13us) server_negotiation.cc:407] Connection header received
0924 03:18:35.642005 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.642009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.642091 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.642200 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:35.644468 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.645196 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.646752 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.647257 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.647422 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.650545 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:35.650550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:35.650553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:35.650582 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:35.653836 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.654596 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.654603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.654607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.654760 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.655198 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.655201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.655203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.655436 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:35.655686 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.655997 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.656183 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":730,"threads_started":1}
I20260924 03:18:35.656653 30098 heartbeater.cc:511] Master 127.14.227.190:37649 requested a full tablet report, sending...
I20260924 03:18:35.658222 29650 ts_manager.cc:194] Registered new tserver with Master: 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541)
I20260924 03:18:35.659235 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:18:35.659286 29650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:38373
I20260924 03:18:35.665866 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.660153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.14.227.190:37649)
0924 03:18:35.660303 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.660309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.660792 (+ 483us) server_negotiation.cc:407] Connection header received
0924 03:18:35.661040 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.661045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.661161 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.661301 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:18:35.661714 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.662453 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.663856 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.664258 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.664381 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.664906 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:18:35.665316 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:18:35.665483 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.665540 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.665615 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:18:35.666697 29650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:18:35.673753 29650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:18:35.697722 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.684721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.14.227.131:36541)
0924 03:18:35.685654 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.685663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.686163 (+ 500us) server_negotiation.cc:407] Connection header received
0924 03:18:35.686480 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.686491 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.686784 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.686958 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:35.688074 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.690129 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.693638 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.695493 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.695767 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.695985 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.696933 (+ 948us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.697151 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":1124,"threads_started":1}
I20260924 03:18:35.698527 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.684130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:36541 (local address 127.0.0.1:37582)
0924 03:18:35.686063 (+ 1933us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:35.686079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:35.686254 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:35.687329 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:35.687339 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:35.687376 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:35.687815 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.687829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.690338 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.690345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.693413 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.693432 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.696214 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.696222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.696493 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.696642 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 03:18:35.698012 (+ 1370us) client_negotiation.cc:240] Negotiation successful
0924 03:18:35.698196 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"thread_start_us":13321,"threads_started":1}
I20260924 03:18:35.702876 30033 tablet_service.cc:1467] Processing CreateTablet for tablet 17d5ec5bf7804816868bb2c7b0cd0a37 (DEFAULT_TABLE table=test-table [id=85bf89d4e5834e3ca7ee45dc41b26e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:35.704272 30033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d5ec5bf7804816868bb2c7b0cd0a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:35.704748 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.685348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:39895 (local address 127.0.0.1:38706)
0924 03:18:35.697186 (+ 11838us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:35.697208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:35.697350 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:35.698719 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:35.698724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:35.698745 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:35.699104 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.699116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.700831 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.700838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.702251 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.702266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.703841 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.703847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.704029 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.704060 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:18:35.704169 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:18:35.704298 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11679,"thread_start_us":785,"threads_started":1}
I20260924 03:18:35.704758 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.685983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.14.227.130:39895)
0924 03:18:35.698156 (+ 12173us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.698165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.698196 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:18:35.698283 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.698294 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.698530 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.698722 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:35.699320 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.700693 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.702421 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.703636 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.703833 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.704004 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.704208 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.704363 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12016,"thread_start_us":12980,"threads_started":1}
I20260924 03:18:35.706871 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.684003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33920)
0924 03:18:35.684921 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:35.684963 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:35.685107 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:35.700128 (+ 15021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:35.700135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:35.700163 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:35.700493 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.700507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.702431 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.702436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.704359 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.704373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.706126 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.706132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.706293 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.706322 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:18:35.706418 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 03:18:35.706552 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":595,"threads_started":1}
I20260924 03:18:35.706943 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.684024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.14.227.129:45839)
0924 03:18:35.699554 (+ 15530us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.699562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.699586 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:18:35.699683 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.699691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.699885 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.700118 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:35.700699 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.702276 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.704548 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.705883 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.706075 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.706241 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.706473 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.706624 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15376,"thread_start_us":15269,"threads_started":1}
I20260924 03:18:35.708276 29902 tablet_service.cc:1467] Processing CreateTablet for tablet 17d5ec5bf7804816868bb2c7b0cd0a37 (DEFAULT_TABLE table=test-table [id=85bf89d4e5834e3ca7ee45dc41b26e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:35.709302 29902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d5ec5bf7804816868bb2c7b0cd0a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:35.713450 29770 tablet_service.cc:1467] Processing CreateTablet for tablet 17d5ec5bf7804816868bb2c7b0cd0a37 (DEFAULT_TABLE table=test-table [id=85bf89d4e5834e3ca7ee45dc41b26e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:35.714762 29770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d5ec5bf7804816868bb2c7b0cd0a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:35.717062 30114 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: Bootstrap starting.
I20260924 03:18:35.720425 30114 tablet_bootstrap.cc:654] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:35.721487 30114 log.cc:824] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:35.723897 30116 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: Bootstrap starting.
I20260924 03:18:35.724426 30114 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: No bootstrap required, opened a new log
I20260924 03:18:35.724689 30114 ts_tablet_manager.cc:1397] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260924 03:18:35.727241 30116 tablet_bootstrap.cc:654] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:35.728561 30117 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Bootstrap starting.
I20260924 03:18:35.729300 30116 log.cc:824] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:35.732710 30117 tablet_bootstrap.cc:654] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:35.734270 30117 log.cc:824] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:35.735997 30114 raft_consensus.cc:348] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.736410 30114 raft_consensus.cc:374] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:35.736510 30114 raft_consensus.cc:729] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9027d7b00f5040d994f1c937e03c9009, State: Initialized, Role: FOLLOWER
I20260924 03:18:35.736877 30116 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: No bootstrap required, opened a new log
I20260924 03:18:35.737110 30116 ts_tablet_manager.cc:1397] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260924 03:18:35.737120 30114 consensus_queue.cc:260] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.737371 30117 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: No bootstrap required, opened a new log
I20260924 03:18:35.737680 30117 ts_tablet_manager.cc:1397] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 03:18:35.742616 30098 heartbeater.cc:503] Master 127.14.227.190:37649 was elected leader, sending a full tablet report...
I20260924 03:18:35.743273 30114 ts_tablet_manager.cc:1428] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: Time spent starting tablet: real 0.018s user 0.010s sys 0.009s
I20260924 03:18:35.745397 30116 raft_consensus.cc:348] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.745759 30116 raft_consensus.cc:374] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:35.745877 30116 raft_consensus.cc:729] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda96edc9be541fc92ee4390495a5390, State: Initialized, Role: FOLLOWER
I20260924 03:18:35.746486 30116 consensus_queue.cc:260] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.748544 30116 ts_tablet_manager.cc:1428] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 03:18:35.749071 29967 heartbeater.cc:503] Master 127.14.227.190:37649 was elected leader, sending a full tablet report...
I20260924 03:18:35.749782 30117 raft_consensus.cc:348] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.750192 30117 raft_consensus.cc:374] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:35.750305 30117 raft_consensus.cc:729] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302188598ba44b5598121d6a96896a30, State: Initialized, Role: FOLLOWER
I20260924 03:18:35.750842 30117 consensus_queue.cc:260] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.751852 29835 heartbeater.cc:503] Master 127.14.227.190:37649 was elected leader, sending a full tablet report...
I20260924 03:18:35.752509 30117 ts_tablet_manager.cc:1428] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Time spent starting tablet: real 0.015s user 0.005s sys 0.007s
W20260924 03:18:35.758246 29836 tablet.cc:2369] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:18:35.758358 30120 raft_consensus.cc:482] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:35.758625 30120 raft_consensus.cc:504] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.760242 30120 leader_election.cc:288] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 302188598ba44b5598121d6a96896a30 (127.14.227.129:45839), cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
I20260924 03:18:35.766461 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.760542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:58235 (local address 127.14.227.129:45839)
0924 03:18:35.760723 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.760731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.761167 (+ 436us) server_negotiation.cc:407] Connection header received
0924 03:18:35.761250 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.761258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.761344 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.761433 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:35.762033 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.762804 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.764756 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.765565 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.765723 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.765794 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.766099 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.766219 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:18:35.766467 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.760562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.14.227.131:58235)
0924 03:18:35.760886 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:35.760904 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:35.761043 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:35.761591 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:35.761595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:35.761614 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:35.761859 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.761871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.763017 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.763022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.764591 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.764602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.765709 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.765715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.765899 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.765934 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:18:35.766081 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 03:18:35.766207 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":22}
I20260924 03:18:35.767426 29790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d5ec5bf7804816868bb2c7b0cd0a37" candidate_uuid: "9027d7b00f5040d994f1c937e03c9009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302188598ba44b5598121d6a96896a30" is_pre_election: true
I20260924 03:18:35.767901 29790 raft_consensus.cc:2454] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9027d7b00f5040d994f1c937e03c9009 in term 0.
I20260924 03:18:35.768558 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.761098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:37119 (local address 127.14.227.130:39895)
0924 03:18:35.761259 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.761265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.761775 (+ 510us) server_negotiation.cc:407] Connection header received
0924 03:18:35.762027 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.762033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.762098 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.762150 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:35.762805 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.763804 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.766242 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.767408 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.767536 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.767599 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.768186 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.768332 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:18:35.768592 29988 leader_election.cc:302] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302188598ba44b5598121d6a96896a30, 9027d7b00f5040d994f1c937e03c9009; no voters:
I20260924 03:18:35.768776 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.760911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:39895 (local address 127.14.227.131:37119)
0924 03:18:35.761675 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:35.761697 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:35.761866 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:35.762336 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:35.762340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:35.762360 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:35.762659 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.762667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.763956 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.763961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.766093 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:35.766106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.767771 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.767776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:35.767964 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.767995 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 03:18:35.768142 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 03:18:35.768258 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":1,"thread_start_us":472,"threads_started":1}
I20260924 03:18:35.769093 30120 raft_consensus.cc:2781] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:18:35.769243 30120 raft_consensus.cc:482] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:18:35.769366 30120 raft_consensus.cc:3037] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:35.769654 29922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d5ec5bf7804816868bb2c7b0cd0a37" candidate_uuid: "9027d7b00f5040d994f1c937e03c9009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda96edc9be541fc92ee4390495a5390" is_pre_election: true
I20260924 03:18:35.770151 29922 raft_consensus.cc:2454] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9027d7b00f5040d994f1c937e03c9009 in term 0.
I20260924 03:18:35.771797 30120 raft_consensus.cc:504] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.772334 30120 leader_election.cc:288] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [CANDIDATE]: Term 1 election: Requested vote from peers 302188598ba44b5598121d6a96896a30 (127.14.227.129:45839), cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
I20260924 03:18:35.772758 29790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d5ec5bf7804816868bb2c7b0cd0a37" candidate_uuid: "9027d7b00f5040d994f1c937e03c9009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "302188598ba44b5598121d6a96896a30"
I20260924 03:18:35.772809 29922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d5ec5bf7804816868bb2c7b0cd0a37" candidate_uuid: "9027d7b00f5040d994f1c937e03c9009" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cda96edc9be541fc92ee4390495a5390"
I20260924 03:18:35.772935 29790 raft_consensus.cc:3037] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:35.773029 29922 raft_consensus.cc:3037] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:35.775480 29790 raft_consensus.cc:2454] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9027d7b00f5040d994f1c937e03c9009 in term 1.
I20260924 03:18:35.775681 29922 raft_consensus.cc:2454] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9027d7b00f5040d994f1c937e03c9009 in term 1.
I20260924 03:18:35.775871 29988 leader_election.cc:302] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 302188598ba44b5598121d6a96896a30, 9027d7b00f5040d994f1c937e03c9009; no voters:
I20260924 03:18:35.776146 30120 raft_consensus.cc:2781] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:35.776516 30120 raft_consensus.cc:686] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 1 LEADER]: Becoming Leader. State: Replica: 9027d7b00f5040d994f1c937e03c9009, State: Running, Role: LEADER
I20260924 03:18:35.776932 30120 consensus_queue.cc:237] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:35.781445 29650 catalog_manager.cc:5495] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131). New cstate: current_term: 1 leader_uuid: "9027d7b00f5040d994f1c937e03c9009" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } health_report { overall_health: UNKNOWN } } }
I20260924 03:18:35.806331 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.801371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.14.227.131:36541)
0924 03:18:35.801539 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.801543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.801609 (+ 66us) server_negotiation.cc:407] Connection header received
0924 03:18:35.801773 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.801777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.801888 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.802064 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:18:35.802363 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.803085 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.804451 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.804867 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.804960 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.805371 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:18:35.805850 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:18:35.806051 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.806091 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.806149 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 03:18:35.813737 29790 raft_consensus.cc:1261] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Refusing update from remote peer 9027d7b00f5040d994f1c937e03c9009: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:35.813879 29922 raft_consensus.cc:1261] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 1 FOLLOWER]: Refusing update from remote peer 9027d7b00f5040d994f1c937e03c9009: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:35.814752 30124 consensus_queue.cc:1035] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [LEADER]: Connected to new peer: Peer: permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:35.815102 30120 consensus_queue.cc:1035] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:35.827646 30129 mvcc.cc:205] Tried to move back new op lower bound from 7332740775163899904 to 7332740775025844224. Current Snapshot: MvccSnapshot[applied={T|T < 7332740775163899904}]
I20260924 03:18:35.837277 29649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:18:35.837612 29649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:18:35.841228 29649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Sending DeleteTablet for 3 replicas of tablet 17d5ec5bf7804816868bb2c7b0cd0a37
I20260924 03:18:35.842332 29770 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d5ec5bf7804816868bb2c7b0cd0a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:35 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:33920
I20260924 03:18:35.842397 30033 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d5ec5bf7804816868bb2c7b0cd0a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:35 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:37582
I20260924 03:18:35.842674 29902 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d5ec5bf7804816868bb2c7b0cd0a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:35 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38706
I20260924 03:18:35.843317 30136 tablet_replica.cc:330] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: stopping tablet replica
I20260924 03:18:35.843572 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 29687
I20260924 03:18:35.843786 30138 tablet_replica.cc:330] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: stopping tablet replica
I20260924 03:18:35.843827 30136 raft_consensus.cc:2229] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:18:35.844282 30138 raft_consensus.cc:2229] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:18:35.844617 30136 raft_consensus.cc:2258] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:35.844763 30138 raft_consensus.cc:2258] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:35.845399 30137 tablet_replica.cc:330] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: stopping tablet replica
I20260924 03:18:35.845822 30137 raft_consensus.cc:2229] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:18:35.846362 30137 raft_consensus.cc:2258] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:35.846935 30136 ts_tablet_manager.cc:1905] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:35.853664 30136 ts_tablet_manager.cc:1918] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:35.853824 30136 log.cc:1196] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/17d5ec5bf7804816868bb2c7b0cd0a37
I20260924 03:18:35.854370 30136 ts_tablet_manager.cc:1939] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 9027d7b00f5040d994f1c937e03c9009: Deleting consensus metadata
I20260924 03:18:35.856549 29634 catalog_manager.cc:4841] TS 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541): tablet 17d5ec5bf7804816868bb2c7b0cd0a37 (table test-table [id=85bf89d4e5834e3ca7ee45dc41b26e3f]) successfully deleted
I20260924 03:18:35.857901 30138 ts_tablet_manager.cc:1905] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:35.859346 30137 ts_tablet_manager.cc:1905] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:35.861490 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.844327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.14.227.129:45839)
0924 03:18:35.844898 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.844906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.844925 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:18:35.844976 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.844980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.845055 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.845151 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:35.847238 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.848306 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.850065 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.850657 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.850783 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.854133 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:35.854148 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:35.854170 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:35.854212 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:35.858082 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.858996 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.859006 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.859009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.859162 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.859660 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.859665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.859668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.860124 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:35.860407 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.860725 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 03:18:35.860959 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260924 03:18:35.862385 29830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:35.865886 30138 ts_tablet_manager.cc:1918] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:35.866092 30138 log.cc:1196] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/17d5ec5bf7804816868bb2c7b0cd0a37
I20260924 03:18:35.866611 30138 ts_tablet_manager.cc:1939] T 17d5ec5bf7804816868bb2c7b0cd0a37 P cda96edc9be541fc92ee4390495a5390: Deleting consensus metadata
I20260924 03:18:35.868312 29636 catalog_manager.cc:4841] TS cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895): tablet 17d5ec5bf7804816868bb2c7b0cd0a37 (table test-table [id=85bf89d4e5834e3ca7ee45dc41b26e3f]) successfully deleted
I20260924 03:18:35.967737 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 29687
I20260924 03:18:35.985632 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 29839
W20260924 03:18:35.985973 29636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.227.129:45839 (error 108)
I20260924 03:18:35.988307 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.987793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33936)
0924 03:18:35.988008 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:35.988124 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:18:35.988593 29636 catalog_manager.cc:4584] TS 302188598ba44b5598121d6a96896a30 (127.14.227.129:45839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17d5ec5bf7804816868bb2c7b0cd0a37: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
I20260924 03:18:36.000538 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:35.986788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.14.227.130:39895)
0924 03:18:35.986979 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:35.986987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:35.987008 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:18:35.987079 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:35.987084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:35.987167 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:35.987271 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:35.989272 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.990074 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.991400 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:35.991811 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:35.991949 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:35.994733 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:35.994756 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:35.994771 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:35.994811 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:35.997606 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.998449 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.998458 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.998462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.998601 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:35.998975 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:35.998981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:35.998984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:35.999353 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:35.999648 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:35.999970 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 03:18:36.000200 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:18:36.001199 29962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:36.042481 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:36.041915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33950)
0924 03:18:36.042156 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:36.042286 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 03:18:36.043599 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:36.043232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33962)
0924 03:18:36.043385 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:36.043465 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 03:18:36.090143 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 29839
I20260924 03:18:36.104372 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:36.103762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33966)
0924 03:18:36.104038 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:36.104153 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 03:18:36.105604 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:36.105177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33974)
0924 03:18:36.105360 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:36.105453 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 03:18:36.106876 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 29970
I20260924 03:18:36.121090 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:36.107603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.14.227.131:36541)
0924 03:18:36.107767 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:36.107775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:36.107798 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:18:36.107998 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:36.108003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:36.108085 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:36.108195 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:36.109441 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:36.110463 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:36.111804 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:36.112278 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:36.112408 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:36.115113 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:36.115127 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:36.115143 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:36.115174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:36.118277 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:36.118970 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:36.118977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:36.118980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:36.119114 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:36.119487 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:36.119494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:36.119497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:36.119855 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:36.120181 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:36.120480 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 03:18:36.120715 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:18:36.121800 30093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:36.186030 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:36.185483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33986)
0924 03:18:36.185693 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:36.185790 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 03:18:36.187059 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:36.186663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:33998)
0924 03:18:36.186891 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:36.186961 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 03:18:36.242262 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 29970
I20260924 03:18:36.262442 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 29617
I20260924 03:18:36.263630 29680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:36.376696 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:36.376021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45839 (local address 127.0.0.1:34004)
0924 03:18:36.376329 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:36.376457 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260924 03:18:36.376906 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 29617
I20260924 03:18:36.397919 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37649
--webserver_interface=127.14.227.190
--webserver_port=33317
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:37649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:36.716151 30144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:36.716482 30144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:36.716639 30144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:36.723738 30144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:36.723877 30144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:36.724020 30144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:36.724112 30144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:36.724185 30144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:36.733629 30144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:37649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=33317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:36.734956 30144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:36.736228 30144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:36.747475 30152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:36.747633 30150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:36.747601 30149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:36.748813 30144 server_base.cc:1034] running on GCE node
I20260924 03:18:36.749594 30144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:36.751025 30144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:36.752254 30144 hybrid_clock.cc:648] HybridClock initialized: now 1790219916752206 us; error 55 us; skew 500 ppm
I20260924 03:18:36.752585 30144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:36 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219916, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:36.761286 30144 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:18:36.762828 30144 webserver.cc:460] Webserver started at http://127.14.227.190:33317/ using document root <none> and password file <none>
I20260924 03:18:36.763512 30144 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:36.763614 30144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:36.769073 30144 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:18:36.771833 30159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:36.773118 30144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 03:18:36.773283 30144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "bc7febfd13fd49248500f83468275e5c"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "82f79e856335159ea3168bbecaf11cc4"
server_key_iv: "a884dcb92173d2922695c7748e8716fe"
server_key_version: "encryptionkey@0"
I20260924 03:18:36.774260 30144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:36.791105 30144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:36.794860 30144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:36.795137 30144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:36.813798 30144 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:37649
I20260924 03:18:36.813834 30210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:37649 every 8 connection(s)
I20260924 03:18:36.815227 30144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:18:36.819218 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 30144
I20260924 03:18:36.820037 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:45839
--local_ip_for_outbound_sockets=127.14.227.129
--tserver_master_addrs=127.14.227.190:37649
--webserver_port=44741
--webserver_interface=127.14.227.129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:18:36.822309 30211 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:18:36.825726 30211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Bootstrap starting.
I20260924 03:18:36.845465 30211 log.cc:824] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:36.860988 30211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:18:36.861596 30211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c: Bootstrap complete.
I20260924 03:18:36.870541 30211 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:36.871383 30211 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7febfd13fd49248500f83468275e5c, State: Initialized, Role: FOLLOWER
I20260924 03:18:36.872014 30211 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:36.872218 30211 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:18:36.872345 30211 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:18:36.872516 30211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:18:36.876216 30211 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:36.876698 30211 leader_election.cc:302] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc7febfd13fd49248500f83468275e5c; no voters:
I20260924 03:18:36.877374 30211 leader_election.cc:288] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:18:36.877504 30215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:18:36.878806 30215 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [term 3 LEADER]: Becoming Leader. State: Replica: bc7febfd13fd49248500f83468275e5c, State: Running, Role: LEADER
I20260924 03:18:36.879549 30211 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:36.879416 30215 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } }
I20260924 03:18:36.881675 30217 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc7febfd13fd49248500f83468275e5c. Latest consensus state: current_term: 3 leader_uuid: "bc7febfd13fd49248500f83468275e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } } }
I20260924 03:18:36.882376 30217 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:36.882107 30216 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bc7febfd13fd49248500f83468275e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc7febfd13fd49248500f83468275e5c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37649 } } }
I20260924 03:18:36.882805 30216 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc7febfd13fd49248500f83468275e5c [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:36.891423 30221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:18:36.901877 30221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:18:36.903846 30221 catalog_manager.cc:1259] Loaded cluster ID: 942aeb49e73c4b86a21a4747be7857ef
I20260924 03:18:36.904017 30221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 03:18:36.906581 30221 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efeb06ce9ad google::LogMessage::Fail() at ??:0
@ 0x7efeb06d188a google::LogMessage::SendToLog() at ??:0
@ 0x7efeb06ce4de google::LogMessage::Flush() at ??:0
@ 0x7efeb06cfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efec6101512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7efec60fe976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efec6186610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7efec6186439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efeb331a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efeb331d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efeb331d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efeb32f8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7efeb65d26db start_thread at ??:0
@ 0x7efeaf56861f clone at ??:0
W20260924 03:18:37.208002 30213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:37.208274 30213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:37.208367 30213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:37.215535 30213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:37.215631 30213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:37.215737 30213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:18:37.225155 30213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:45839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=44741
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:37.226423 30213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:37.227730 30213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:37.240367 30253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:37.240451 30252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:37.240471 30255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:37.241286 30213 server_base.cc:1034] running on GCE node
I20260924 03:18:37.241858 30213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:37.242754 30213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:37.243986 30213 hybrid_clock.cc:648] HybridClock initialized: now 1790219917243950 us; error 51 us; skew 500 ppm
I20260924 03:18:37.244304 30213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:37 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219917, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:37.247457 30213 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:18:37.248884 30213 webserver.cc:460] Webserver started at http://127.14.227.129:44741/ using document root <none> and password file <none>
I20260924 03:18:37.249528 30213 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:37.249675 30213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:37.254843 30213 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:18:37.257714 30262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:37.259071 30213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:18:37.259260 30213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "302188598ba44b5598121d6a96896a30"
format_stamp: "Formatted at 2026-09-24 03:18:31 on dist-test-slave-vd0p"
server_key: "56c7fa95c9ed525d238f4e1b313b5a39"
server_key_iv: "6d36e5438bb6132bf9cd248fe4646458"
server_key_version: "encryptionkey@0"
I20260924 03:18:37.260185 30213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:37.282069 30213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:37.286250 30213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:37.286520 30213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:37.287456 30213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:37.289106 30269 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 03:18:37.294462 30213 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:18:37.294620 30213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 03:18:37.294788 30213 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:18:37.298070 30213 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:18:37.298192 30213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.004s sys 0.000s
I20260924 03:18:37.298475 30269 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Bootstrap starting.
I20260924 03:18:37.306228 30269 log.cc:824] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:37.319334 30269 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:18:37.319907 30269 tablet_bootstrap.cc:492] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Bootstrap complete.
I20260924 03:18:37.320320 30269 ts_tablet_manager.cc:1397] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.003s
I20260924 03:18:37.330081 30269 raft_consensus.cc:348] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:37.331583 30269 raft_consensus.cc:729] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302188598ba44b5598121d6a96896a30, State: Initialized, Role: FOLLOWER
I20260924 03:18:37.332357 30269 consensus_queue.cc:260] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:37.333890 30269 ts_tablet_manager.cc:1428] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260924 03:18:37.344381 30213 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:45839
I20260924 03:18:37.344411 30377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:45839 every 8 connection(s)
I20260924 03:18:37.345703 30213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:18:37.350473 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:37.348654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:34503)
0924 03:18:37.349381 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:37.349607 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":297,"threads_started":1}
W20260924 03:18:37.351054 30378 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:37649 (0 consecutive failures): Network error: Failed to ping master at 127.14.227.190:37649: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
I20260924 03:18:37.351784 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:37.351511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:33559)
0924 03:18:37.351633 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:37.351692 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:18:37.352569 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:37.352261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:46713)
0924 03:18:37.352382 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:37.352446 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 03:18:37.352839 30378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:18:37.353122 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 30213
I20260924 03:18:37.353775 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:39895
--local_ip_for_outbound_sockets=127.14.227.130
--tserver_master_addrs=127.14.227.190:37649
--webserver_port=37271
--webserver_interface=127.14.227.130
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:37.684734 30381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:37.685020 30381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:37.685185 30381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:37.692028 30381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:37.692122 30381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:37.692329 30381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:18:37.701511 30381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:39895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=37271
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:37.702755 30381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:37.704443 30381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:37.718050 30387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:37.717974 30386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:37.718083 30389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:37.718740 30381 server_base.cc:1034] running on GCE node
I20260924 03:18:37.719309 30381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:37.720196 30381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:37.721418 30381 hybrid_clock.cc:648] HybridClock initialized: now 1790219917721374 us; error 56 us; skew 500 ppm
I20260924 03:18:37.721733 30381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:37 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219917, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:37.724851 30381 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:18:37.726349 30381 webserver.cc:460] Webserver started at http://127.14.227.130:37271/ using document root <none> and password file <none>
I20260924 03:18:37.727015 30381 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:37.727159 30381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:37.732303 30381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:18:37.737181 30396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:37.738571 30381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:18:37.738741 30381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "cda96edc9be541fc92ee4390495a5390"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "d9de3c03390d91bb287110f696fca4fd"
server_key_iv: "4f546d18b16aac567e2b1cd7479ed01a"
server_key_version: "encryptionkey@0"
I20260924 03:18:37.739665 30381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:37.763417 30381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:37.767580 30381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:37.767808 30381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:37.768606 30381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:37.769826 30381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:37.769912 30381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:37.770035 30381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:37.770097 30381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:37.801899 30381 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:39895
I20260924 03:18:37.802021 30508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:39895 every 8 connection(s)
I20260924 03:18:37.803359 30381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:18:37.807546 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 30381
I20260924 03:18:37.808427 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:36541
--local_ip_for_outbound_sockets=127.14.227.131
--tserver_master_addrs=127.14.227.190:37649
--webserver_port=37527
--webserver_interface=127.14.227.131
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:18:37.808719 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:37.806387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:36813)
0924 03:18:37.807179 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:37.807506 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":273,"threads_started":1}
W20260924 03:18:37.809449 30509 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:37649 (0 consecutive failures): Network error: Failed to ping master at 127.14.227.190:37649: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
I20260924 03:18:37.810490 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:37.810026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:45673)
0924 03:18:37.810234 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:37.810337 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 03:18:37.811486 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:37.811038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:50181)
0924 03:18:37.811225 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:37.811329 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 03:18:37.811766 30509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:18:38.131273 30512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:38.131521 30512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:38.131681 30512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:38.138417 30512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:38.138504 30512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:38.138659 30512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:18:38.148092 30512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:36541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=37527
--tserver_master_addrs=127.14.227.190:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:38.149415 30512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:38.150717 30512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:38.163360 30520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:38.163457 30517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:38.163439 30518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:38.164638 30512 server_base.cc:1034] running on GCE node
I20260924 03:18:38.165237 30512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:38.166167 30512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:38.167400 30512 hybrid_clock.cc:648] HybridClock initialized: now 1790219918167369 us; error 58 us; skew 500 ppm
I20260924 03:18:38.167701 30512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:38 dist-test-slave-vd0p krb5kdc[29067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219918, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:38.170622 30512 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:18:38.171937 30512 webserver.cc:460] Webserver started at http://127.14.227.131:37527/ using document root <none> and password file <none>
I20260924 03:18:38.172571 30512 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:38.172714 30512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:38.177790 30512 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:18:38.180371 30527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:38.181608 30512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:18:38.181782 30512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "9027d7b00f5040d994f1c937e03c9009"
format_stamp: "Formatted at 2026-09-24 03:18:32 on dist-test-slave-vd0p"
server_key: "ce04de251268cd1d5e07c80092825dfb"
server_key_iv: "5faaa4f0a6d331a675482c617982b8f6"
server_key_version: "encryptionkey@0"
I20260924 03:18:38.182791 30512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:38.204010 30512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:38.207854 30512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:38.208135 30512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:38.209112 30512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:38.210429 30512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:38.210543 30512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:38.210678 30512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:38.210770 30512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:38.244125 30512 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:36541
I20260924 03:18:38.244283 30639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:36541 every 8 connection(s)
I20260924 03:18:38.245569 30512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:18:38.250095 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 30512
I20260924 03:18:38.250656 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.248649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:34497)
0924 03:18:38.249448 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:38.249701 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":275,"threads_started":1}
W20260924 03:18:38.251471 30640 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:37649 (0 consecutive failures): Network error: Failed to ping master at 127.14.227.190:37649: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
I20260924 03:18:38.252358 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.252036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:60357)
0924 03:18:38.252175 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:38.252253 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:18:38.253181 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.252872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:36045)
0924 03:18:38.252990 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:38.253068 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 03:18:38.253458 30640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:18:38.354023 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.353289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:39925)
0924 03:18:38.353717 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:38.353803 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":213,"threads_started":1}
I20260924 03:18:38.505235 30645 raft_consensus.cc:482] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:38.505590 30645 raft_consensus.cc:504] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:38.507381 30645 leader_election.cc:288] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541), cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
Sep 24 03:18:38 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219917, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
Sep 24 03:18:38 dist-test-slave-vd0p krb5kdc[29067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219917, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:18:38.528590 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.507619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:36541 (local address 127.14.227.129:43827)
0924 03:18:38.507958 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:38.508007 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:38.508645 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:38.512276 (+ 3631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:38.512288 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:38.513074 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:38.513696 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:38.513713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:38.515579 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:38.515582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:38.517063 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:38.517075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:38.518072 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:38.518082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:38.518336 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:38.519122 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:38.519139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:38.521783 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:38.525639 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:38.525643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:38.525646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:38.526087 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:38.526539 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:38.526544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:38.526548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:38.526693 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:38.527468 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:38.527473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:38.527609 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 03:18:38.527765 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 03:18:38.528146 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":82}
I20260924 03:18:38.528652 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.508068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:34845 (local address 127.14.227.130:39895)
0924 03:18:38.510099 (+ 2031us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:38.510109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:38.510138 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:18:38.511418 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:38.511425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:38.511831 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:38.512157 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:38.513912 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:38.515289 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:38.517203 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:38.517839 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:38.518062 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:38.521696 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:38.521706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:38.521732 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:38.521792 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:38.524741 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:38.525740 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:38.525747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:38.525752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:38.525897 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:38.526404 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:38.526409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:38.526413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:38.526920 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:38.527108 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:38.527839 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 03:18:38.528227 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1853,"thread_start_us":1763,"threads_started":1}
I20260924 03:18:38.528795 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.507909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:43827 (local address 127.14.227.131:36541)
0924 03:18:38.509934 (+ 2025us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:38.509995 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:38.510037 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:18:38.511413 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:38.511420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:38.511813 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:38.512080 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:38.513895 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:38.515385 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:38.517200 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:38.517843 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:38.518053 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:38.522063 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:38.522070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:38.522081 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:38.522115 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:38.525199 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:38.526218 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:38.526248 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:38.526253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:38.526419 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:38.526825 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:38.526832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:38.526839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:38.527335 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:38.527515 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:38.528007 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 03:18:38.528383 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1844,"thread_start_us":1662,"threads_started":1}
I20260924 03:18:38.528702 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.507979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:39895 (local address 127.14.227.129:34845)
0924 03:18:38.509531 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:38.509566 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:38.509661 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:38.512498 (+ 2837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:38.512501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:38.513085 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:38.513709 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:38.513726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:38.515482 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:38.515491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:38.517065 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:38.517076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:38.518104 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:38.518111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:38.518337 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:38.519053 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:38.519078 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:38.521401 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:38.524879 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:38.524888 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:38.524902 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:38.525552 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:38.526068 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:38.526073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:38.526077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:38.526265 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:38.527062 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:38.527069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:38.527354 (+ 285us) client_negotiation.cc:769] Sending connection context
0924 03:18:38.527626 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 03:18:38.528295 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"mutex_wait_us":371,"spinlock_wait_cycles":27264,"thread_start_us":1164,"threads_started":1}
W20260924 03:18:38.531960 30265 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
W20260924 03:18:38.532217 30263 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
I20260924 03:18:38.532377 30263 leader_election.cc:302] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 302188598ba44b5598121d6a96896a30; no voters: 9027d7b00f5040d994f1c937e03c9009, cda96edc9be541fc92ee4390495a5390
I20260924 03:18:38.532759 30645 raft_consensus.cc:2726] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:18:38.813099 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:38.812245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:57025)
0924 03:18:38.812704 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:38.812850 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":235,"threads_started":1}
I20260924 03:18:39.255205 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:39.254244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:55367)
0924 03:18:39.254846 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:39.254965 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":327,"threads_started":1}
I20260924 03:18:39.355710 30651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:39.354926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:34223)
0924 03:18:39.355409 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:39.355492 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":271,"threads_started":1}
I20260924 03:18:39.815008 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:39.814179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:40257)
0924 03:18:39.814688 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:39.814796 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":299,"threads_started":1}
I20260924 03:18:40.256836 30653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:40.256181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:55057)
0924 03:18:40.256590 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:40.256664 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":216,"threads_started":1}
I20260924 03:18:40.357224 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:40.356532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:58483)
0924 03:18:40.356987 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:40.357060 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":235,"threads_started":1}
I20260924 03:18:40.658378 30655 raft_consensus.cc:482] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:40.658568 30655 raft_consensus.cc:504] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:40.659214 30655 leader_election.cc:288] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541), cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
W20260924 03:18:40.660197 30263 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
W20260924 03:18:40.660416 30265 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
I20260924 03:18:40.660584 30265 leader_election.cc:302] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 302188598ba44b5598121d6a96896a30; no voters: 9027d7b00f5040d994f1c937e03c9009, cda96edc9be541fc92ee4390495a5390
I20260924 03:18:40.660846 30655 raft_consensus.cc:2726] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:18:40.816497 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:40.815813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:39635)
0924 03:18:40.816229 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:40.816306 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":227,"threads_started":1}
I20260924 03:18:41.258428 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:41.257671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:46575)
0924 03:18:41.258164 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:41.258250 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":232,"threads_started":1}
I20260924 03:18:41.358768 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:41.358136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:55635)
0924 03:18:41.358544 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:41.358616 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":204,"threads_started":1}
I20260924 03:18:41.818068 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:41.817315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:49463)
0924 03:18:41.817726 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:41.817833 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":205,"threads_started":1}
I20260924 03:18:42.259953 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:42.259267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:42493)
0924 03:18:42.259689 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:42.259764 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":220,"threads_started":1}
I20260924 03:18:42.358305 30661 raft_consensus.cc:482] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:42.358453 30661 raft_consensus.cc:504] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:42.359036 30661 leader_election.cc:288] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541), cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
W20260924 03:18:42.359912 30265 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
W20260924 03:18:42.360055 30263 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
I20260924 03:18:42.360234 30263 leader_election.cc:302] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 302188598ba44b5598121d6a96896a30; no voters: 9027d7b00f5040d994f1c937e03c9009, cda96edc9be541fc92ee4390495a5390
I20260924 03:18:42.360472 30661 raft_consensus.cc:2726] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:18:42.360682 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:42.359603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:58871)
0924 03:18:42.360322 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:42.360437 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":369,"threads_started":1}
I20260924 03:18:42.819865 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:42.818964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:40593)
0924 03:18:42.819581 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:42.819689 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":359,"threads_started":1}
I20260924 03:18:43.261660 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:43.260996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:49999)
0924 03:18:43.261419 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:43.261496 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":226,"threads_started":1}
I20260924 03:18:43.362156 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:43.361458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:44411)
0924 03:18:43.361872 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:43.362011 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260924 03:18:43.821548 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:43.820848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:58023)
0924 03:18:43.821265 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:43.821372 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260924 03:18:44.157836 30667 raft_consensus.cc:482] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:44.158118 30667 raft_consensus.cc:504] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:44.158792 30667 leader_election.cc:288] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541), cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
W20260924 03:18:44.159741 30265 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
W20260924 03:18:44.159917 30263 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
I20260924 03:18:44.160105 30263 leader_election.cc:302] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 302188598ba44b5598121d6a96896a30; no voters: 9027d7b00f5040d994f1c937e03c9009, cda96edc9be541fc92ee4390495a5390
I20260924 03:18:44.160405 30667 raft_consensus.cc:2726] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:18:44.263319 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:44.262661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:52493)
0924 03:18:44.263067 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:44.263141 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":222,"threads_started":1}
I20260924 03:18:44.363718 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:44.362946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:55985)
0924 03:18:44.363447 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:44.363520 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":255,"threads_started":1}
I20260924 03:18:44.823210 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:44.822469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:35987)
0924 03:18:44.822962 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:44.823040 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":220,"threads_started":1}
I20260924 03:18:45.264951 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:45.264189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:56701)
0924 03:18:45.264655 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:45.264741 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
I20260924 03:18:45.365237 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:45.364506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:44513)
0924 03:18:45.364992 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:45.365073 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":281,"threads_started":1}
I20260924 03:18:45.824739 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:45.824080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:50849)
0924 03:18:45.824481 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:45.824562 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":215,"threads_started":1}
I20260924 03:18:46.266607 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:46.265813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:53711)
0924 03:18:46.266301 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:46.266391 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":276,"threads_started":1}
I20260924 03:18:46.366787 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:46.366068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:40361)
0924 03:18:46.366546 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:46.366623 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":227,"threads_started":1}
I20260924 03:18:46.826500 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:46.825752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:46377)
0924 03:18:46.826253 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:46.826342 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":240,"threads_started":1}
I20260924 03:18:47.268263 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:47.267539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:42779)
0924 03:18:47.267993 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:47.268089 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260924 03:18:47.368290 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:47.367602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:34359)
0924 03:18:47.368041 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:47.368128 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":247,"threads_started":1}
I20260924 03:18:47.827992 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:47.827319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:53633)
0924 03:18:47.827733 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:47.827825 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
I20260924 03:18:48.269738 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:48.269068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:37417)
0924 03:18:48.269471 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:48.269549 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":197,"threads_started":1}
I20260924 03:18:48.369712 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:48.369057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:43949)
0924 03:18:48.369456 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:48.369530 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":222,"threads_started":1}
I20260924 03:18:48.829584 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:48.828812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:41365)
0924 03:18:48.829305 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:48.829398 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
I20260924 03:18:48.835952 30683 raft_consensus.cc:482] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:48.836164 30683 raft_consensus.cc:504] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "302188598ba44b5598121d6a96896a30" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45839 } } peers { permanent_uuid: "9027d7b00f5040d994f1c937e03c9009" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36541 } } peers { permanent_uuid: "cda96edc9be541fc92ee4390495a5390" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39895 } }
I20260924 03:18:48.836876 30683 leader_election.cc:288] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541), cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895)
W20260924 03:18:48.837808 30263 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9027d7b00f5040d994f1c937e03c9009 (127.14.227.131:36541): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
W20260924 03:18:48.838001 30265 leader_election.cc:341] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cda96edc9be541fc92ee4390495a5390 (127.14.227.130:39895): Not found: Tablet not found: 17d5ec5bf7804816868bb2c7b0cd0a37
I20260924 03:18:48.838125 30265 leader_election.cc:302] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 302188598ba44b5598121d6a96896a30; no voters: 9027d7b00f5040d994f1c937e03c9009, cda96edc9be541fc92ee4390495a5390
I20260924 03:18:48.838346 30683 raft_consensus.cc:2726] T 17d5ec5bf7804816868bb2c7b0cd0a37 P 302188598ba44b5598121d6a96896a30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:18:49.271180 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:49.270553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:54861)
0924 03:18:49.270942 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:49.271014 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":207,"threads_started":1}
I20260924 03:18:49.371137 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:49.370536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:53707)
0924 03:18:49.370913 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:49.370992 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":207,"threads_started":1}
I20260924 03:18:49.831125 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:49.830438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:44119)
0924 03:18:49.830864 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:49.830952 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":223,"threads_started":1}
I20260924 03:18:50.272810 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:50.272163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:42515)
0924 03:18:50.272574 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:50.272650 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":225,"threads_started":1}
I20260924 03:18:50.372819 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:50.372018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:41017)
0924 03:18:50.372554 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:50.372657 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":265,"threads_started":1}
I20260924 03:18:50.832949 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:50.832127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:54927)
0924 03:18:50.832582 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:50.832739 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":228,"threads_started":1}
I20260924 03:18:51.274296 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:51.273615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:42121)
0924 03:18:51.274032 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:51.274117 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260924 03:18:51.374377 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:51.373608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:36573)
0924 03:18:51.374094 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:51.374178 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":236,"threads_started":1}
I20260924 03:18:51.834698 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:51.833854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:41575)
0924 03:18:51.834341 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:51.834455 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260924 03:18:52.275849 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:52.275144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:49843)
0924 03:18:52.275582 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:52.275661 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":252,"threads_started":1}
I20260924 03:18:52.376077 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:52.375407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.129:57295)
0924 03:18:52.375826 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:52.375906 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":203,"threads_started":1}
I20260924 03:18:52.836580 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:52.835925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.130:52717)
0924 03:18:52.836333 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:52.836429 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":212,"threads_started":1}
I20260924 03:18:53.253937 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 30213
I20260924 03:18:53.274493 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:53.255138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.14.227.129:45839)
0924 03:18:53.255707 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:53.255717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:53.255753 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:18:53.255882 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:53.255894 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:53.256144 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:53.256273 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:53.257643 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:53.258640 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:53.259853 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:53.260392 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:53.260493 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:53.269134 (+ 8641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:53.269145 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:53.269160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:53.269197 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:53.271913 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:53.272561 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:53.272567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:53.272570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:53.272694 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:53.273003 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:53.273006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:53.273008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:53.273344 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:53.273557 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:53.273894 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 03:18:53.274188 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"spinlock_wait_cycles":8064,"thread_start_us":342,"threads_started":1}
I20260924 03:18:53.275775 30373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:53.277822 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:53.276963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37649 (local address 127.14.227.131:48523)
0924 03:18:53.277466 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:53.277626 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":232,"threads_started":1}
I20260924 03:18:53.363504 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 30213
I20260924 03:18:53.380898 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 30381
I20260924 03:18:53.397589 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:53.381655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.14.227.130:39895)
0924 03:18:53.382160 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:53.382164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:53.382179 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:18:53.382230 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:53.382234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:53.382288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:53.382372 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:53.383605 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:53.384367 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:53.385488 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:53.386098 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:53.386191 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:53.390224 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:53.390232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:53.390237 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:53.390279 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:53.394732 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:53.395663 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:53.395671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:53.395677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:53.395872 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:53.396215 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:53.396218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:53.396221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:53.396498 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:53.396748 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:53.396929 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:18:53.397108 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":267,"threads_started":1}
I20260924 03:18:53.398377 30504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:53.480826 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 30381
I20260924 03:18:53.497810 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 30512
I20260924 03:18:53.510905 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:53.498737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.14.227.131:36541)
0924 03:18:53.499303 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:53.499308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:53.499324 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:18:53.499394 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:53.499398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:53.499455 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:53.499545 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:53.500834 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:53.501684 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:53.502965 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:53.503379 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:53.503469 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:53.506107 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:53.506111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:53.506115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:53.506143 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:53.508648 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:53.509293 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:53.509296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:53.509298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:53.509415 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:53.509716 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:53.509719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:53.509721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:53.510013 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:53.510230 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:53.510453 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 03:18:53.510608 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":324,"threads_started":1}
I20260924 03:18:53.511585 30635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:53.593175 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 30512
2026-09-24T03:18:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23916 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:18:53 dist-test-slave-vd0p krb5kdc[30705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:18:53 dist-test-slave-vd0p krb5kdc[30705](info): set up 2 sockets
Sep 24 03:18:53 dist-test-slave-vd0p krb5kdc[30705](info): commencing operation
krb5kdc: starting...
W20260924 03:18:54.686497 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:18:54 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:18:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:18:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:18:54.907435 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38957
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:55.221189 30718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:55.221468 30718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:55.221560 30718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:55.228688 30718 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:18:55.228787 30718 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:18:55.228824 30718 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:18:55.228875 30718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:18:55.228940 30718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:55.229001 30718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:55.229045 30718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:18:55.229092 30718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:18:55.239409 30718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:55.240618 30718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:55.241951 30718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:55.254278 30726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:55.254323 30724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:55.254653 30723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:55.255612 30718 server_base.cc:1034] running on GCE node
I20260924 03:18:55.256424 30718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:55.257874 30718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:55.259172 30718 hybrid_clock.cc:648] HybridClock initialized: now 1790219935259098 us; error 32 us; skew 500 ppm
I20260924 03:18:55.259541 30718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:55 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219935, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:55.262814 30718 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:18:55.264276 30718 webserver.cc:460] Webserver started at http://127.14.227.190:33063/ using document root <none> and password file <none>
I20260924 03:18:55.264974 30718 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:55.265108 30718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:55.265450 30718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:55.268417 30718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "2df5d208da25487c88c92b6f54032e20"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "11c92750478807823acc6d510333e378"
server_key_iv: "9fb3d90d06d61d2bed94128385e10fb7"
server_key_version: "encryptionkey@0"
I20260924 03:18:55.269177 30718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "2df5d208da25487c88c92b6f54032e20"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "11c92750478807823acc6d510333e378"
server_key_iv: "9fb3d90d06d61d2bed94128385e10fb7"
server_key_version: "encryptionkey@0"
I20260924 03:18:55.274124 30718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:18:55.277339 30733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:55.278620 30718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:18:55.278791 30718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "2df5d208da25487c88c92b6f54032e20"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "11c92750478807823acc6d510333e378"
server_key_iv: "9fb3d90d06d61d2bed94128385e10fb7"
server_key_version: "encryptionkey@0"
I20260924 03:18:55.278990 30718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:55.302229 30718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:55.306017 30718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:55.306285 30718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:55.324342 30718 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:38957
I20260924 03:18:55.324359 30784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:38957 every 8 connection(s)
I20260924 03:18:55.325635 30718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:18:55.327051 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 30718
I20260924 03:18:55.327278 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:18:55.328148 15246 external_mini_cluster.cc:1428] Setting key 3be30d7a6da22da810e6477b2919c952
I20260924 03:18:55.330452 30785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:18:55 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:55.343618 30785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Bootstrap starting.
I20260924 03:18:55.346695 30785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:55.347711 30785 log.cc:824] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:55.350515 30785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: No bootstrap required, opened a new log
I20260924 03:18:55.351058 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:55.330247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.14.227.190:38957)
0924 03:18:55.331055 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:55.331066 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:55.331109 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:18:55.332720 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:55.332740 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:55.333089 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:55.333405 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:55.335186 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:55.336487 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:55.338370 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:55.338997 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:55.339120 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:55.343767 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:55.343778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:55.343797 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:55.343842 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:55.346963 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:55.347706 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:55.347712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:55.347718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:55.347910 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:55.348316 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:55.348321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:55.348325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:55.348857 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:55.349159 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:55.349594 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 03:18:55.350063 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":418,"threads_started":1}
I20260924 03:18:55.358304 30785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:18:55.358649 30785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:55.358758 30785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df5d208da25487c88c92b6f54032e20, State: Initialized, Role: FOLLOWER
I20260924 03:18:55.359305 30785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:18:55.359531 30785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:18:55.359663 30785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:18:55.359824 30785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:55.362280 30785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:18:55.362718 30785 leader_election.cc:302] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2df5d208da25487c88c92b6f54032e20; no voters:
I20260924 03:18:55.363415 30785 leader_election.cc:288] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:18:55.363523 30790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:55.364233 30790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 1 LEADER]: Becoming Leader. State: Replica: 2df5d208da25487c88c92b6f54032e20, State: Running, Role: LEADER
I20260924 03:18:55.364799 30790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:18:55.365374 30785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:18:55.367713 30792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2df5d208da25487c88c92b6f54032e20. Latest consensus state: current_term: 1 leader_uuid: "2df5d208da25487c88c92b6f54032e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } } }
I20260924 03:18:55.368176 30792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: This master's current role is: LEADER
I20260924 03:18:55.368700 30791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2df5d208da25487c88c92b6f54032e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } } }
I20260924 03:18:55.369055 30791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: This master's current role is: LEADER
W20260924 03:18:55.377815 30806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:18:55.377941 30806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:18:55.378132 30797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:18:55.379673 30797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:18:55.387612 30797 catalog_manager.cc:1347] Generated new cluster ID: 6c8b5d4ca75f46848d2bc4a8e9b0b7e4
I20260924 03:18:55.387800 30797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:18:55.401193 30797 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:18:55.402812 30797 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:18:55.409941 30797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Generated new TSK 0
I20260924 03:18:55.410717 30797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:18:55.481892 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38957
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:55.844774 30811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:55.845032 30811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:55.845121 30811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:55.852424 30811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:55.852525 30811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:55.852743 30811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:18:55.862373 30811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:55.863601 30811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:55.864897 30811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:55.877461 30816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:55.877643 30817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:55.878137 30819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:55.878176 30811 server_base.cc:1034] running on GCE node
I20260924 03:18:55.878800 30811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:55.879673 30811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:55.880896 30811 hybrid_clock.cc:648] HybridClock initialized: now 1790219935880858 us; error 51 us; skew 500 ppm
I20260924 03:18:55.881193 30811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:55 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219935, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:55.884294 30811 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:18:55.885588 30811 webserver.cc:460] Webserver started at http://127.14.227.129:41537/ using document root <none> and password file <none>
I20260924 03:18:55.886304 30811 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:55.886445 30811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:55.886735 30811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:55.889503 30811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "87da36cd174d4ec1af9b20b7d0f51271"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "6b93363eee933cfe654cd9f24c4bbe1b"
server_key_iv: "a97c0d91241a6cb777886adc81899d71"
server_key_version: "encryptionkey@0"
I20260924 03:18:55.890275 30811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "87da36cd174d4ec1af9b20b7d0f51271"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "6b93363eee933cfe654cd9f24c4bbe1b"
server_key_iv: "a97c0d91241a6cb777886adc81899d71"
server_key_version: "encryptionkey@0"
I20260924 03:18:55.895112 30811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 03:18:55.898234 30826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:55.899436 30811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 03:18:55.899603 30811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "87da36cd174d4ec1af9b20b7d0f51271"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "6b93363eee933cfe654cd9f24c4bbe1b"
server_key_iv: "a97c0d91241a6cb777886adc81899d71"
server_key_version: "encryptionkey@0"
I20260924 03:18:55.899789 30811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:55.919085 30811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:55.922757 30811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:55.923039 30811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:55.923928 30811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:55.925174 30811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:55.925284 30811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:55.925433 30811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:55.925506 30811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:55.958213 30811 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:45663
I20260924 03:18:55.958266 30938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:45663 every 8 connection(s)
I20260924 03:18:55.959486 30811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:18:55.969168 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 30811
I20260924 03:18:55.969385 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:18:55.970144 15246 external_mini_cluster.cc:1428] Setting key 41b91c14c4b916d44f66f3d866619431
Sep 24 03:18:55 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219935, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:55.979601 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:55.962018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:51383 (local address 127.14.227.190:38957)
0924 03:18:55.962494 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:55.962498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:55.963314 (+ 816us) server_negotiation.cc:407] Connection header received
0924 03:18:55.965159 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:55.965164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:55.965260 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:55.965378 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:55.966953 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:55.967682 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:55.969514 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:55.969928 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:55.970095 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:55.974002 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:55.974012 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:55.974016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:55.974046 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:55.976583 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:55.977304 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:55.977311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:55.977317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:55.977444 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:55.977864 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:55.977870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:55.977873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:55.978190 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:55.978352 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:55.979012 (+ 660us) server_negotiation.cc:299] Negotiation successful
0924 03:18:55.979236 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260924 03:18:55.980584 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:55.962470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:51383)
0924 03:18:55.963149 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:55.963186 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:55.964869 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:55.965616 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:55.965624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:55.966217 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:55.966758 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:55.966775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:55.967843 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:55.967849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:55.969334 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:55.969350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:55.970207 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:55.970216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:55.970510 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:55.971329 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:55.971355 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:55.973741 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:55.976737 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:55.976745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:55.976761 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:55.977172 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:55.977557 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:55.977562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:55.977566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:55.977754 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:55.978318 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:55.978325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:55.978658 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 03:18:55.978928 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 03:18:55.979473 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260924 03:18:55.982319 30939 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38957
I20260924 03:18:55.982714 30939 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:55.983575 30939 heartbeater.cc:511] Master 127.14.227.190:38957 requested a full tablet report, sending...
I20260924 03:18:55.985929 30750 ts_manager.cc:194] Registered new tserver with Master: 87da36cd174d4ec1af9b20b7d0f51271 (127.14.227.129:45663)
I20260924 03:18:55.988118 30750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:51383
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:18:56.029212 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38957
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:56.341652 30945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:56.341924 30945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:56.342101 30945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:56.349079 30945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:56.349186 30945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:56.349399 30945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:18:56.358776 30945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:56.359963 30945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:56.361167 30945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:56.373507 30951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:56.373633 30950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:56.373608 30953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:56.374328 30945 server_base.cc:1034] running on GCE node
I20260924 03:18:56.374818 30945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:56.375609 30945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:56.376801 30945 hybrid_clock.cc:648] HybridClock initialized: now 1790219936376765 us; error 27 us; skew 500 ppm
I20260924 03:18:56.377099 30945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:56 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219936, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:56.380201 30945 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:18:56.381484 30945 webserver.cc:460] Webserver started at http://127.14.227.130:37699/ using document root <none> and password file <none>
I20260924 03:18:56.382197 30945 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:56.382381 30945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:56.382715 30945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:56.385380 30945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "8c603e77e6a14db6883a712ee08493af"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "42d97d5b2d44b613e983f195ab5e118b"
server_key_iv: "e3dfd539934bba738eeb791b33d4732f"
server_key_version: "encryptionkey@0"
I20260924 03:18:56.386163 30945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "8c603e77e6a14db6883a712ee08493af"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "42d97d5b2d44b613e983f195ab5e118b"
server_key_iv: "e3dfd539934bba738eeb791b33d4732f"
server_key_version: "encryptionkey@0"
I20260924 03:18:56.394870 30945 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260924 03:18:56.401353 30960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:56.403395 30945 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 03:18:56.403609 30945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "8c603e77e6a14db6883a712ee08493af"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "42d97d5b2d44b613e983f195ab5e118b"
server_key_iv: "e3dfd539934bba738eeb791b33d4732f"
server_key_version: "encryptionkey@0"
I20260924 03:18:56.403880 30945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:56.428870 30945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:56.433632 30945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:56.433917 30945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:56.434939 30945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:56.436360 30945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:56.436466 30945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:56.436586 30945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:56.436642 30945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:56.472388 30945 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:35029
I20260924 03:18:56.472481 31072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:35029 every 8 connection(s)
I20260924 03:18:56.474002 30945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:18:56.480898 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 30945
I20260924 03:18:56.481241 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:18:56.481802 15246 external_mini_cluster.cc:1428] Setting key 68f35771076e9c39c3a9dbbf81743ba1
Sep 24 03:18:56 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219936, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:56.494427 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:56.476560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:59799 (local address 127.14.227.190:38957)
0924 03:18:56.476766 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:56.476773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:56.478079 (+ 1306us) server_negotiation.cc:407] Connection header received
0924 03:18:56.479810 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:56.479814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:56.479882 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:56.479972 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:56.481546 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:56.482587 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:56.484367 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:56.484800 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:56.484930 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:56.488302 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:56.488310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:56.488319 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:56.488348 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:56.491436 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:56.492149 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:56.492155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:56.492158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:56.492288 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:56.492698 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:56.492704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:56.492707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:56.492985 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:56.493139 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:56.493842 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 03:18:56.494098 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:18:56.495399 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:56.477138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:59799)
0924 03:18:56.477880 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:56.477922 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:56.479560 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:56.480165 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:56.480173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:56.480685 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:56.481363 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:56.481390 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:56.482737 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:56.482743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:56.484201 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:56.484214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:56.485018 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:56.485025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:56.485243 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:56.485910 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:56.485937 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:56.488086 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:56.491584 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:56.491593 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:56.491610 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:56.492019 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:56.492396 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:56.492402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:56.492406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:56.492587 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:56.493115 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:56.493123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:56.493466 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 03:18:56.493744 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 03:18:56.494291 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":261,"threads_started":1}
I20260924 03:18:56.496869 31073 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38957
I20260924 03:18:56.497195 31073 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:56.498075 31073 heartbeater.cc:511] Master 127.14.227.190:38957 requested a full tablet report, sending...
I20260924 03:18:56.499670 30750 ts_manager.cc:194] Registered new tserver with Master: 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029)
I20260924 03:18:56.501000 30750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:59799
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:18:56.535391 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38957
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:18:56.850347 31078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:18:56.850608 31078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:18:56.850778 31078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:18:56.857789 31078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:18:56.857887 31078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:18:56.858084 31078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:18:56.867708 31078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:18:56.868928 31078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:18:56.870220 31078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:18:56.883265 31086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:56.883198 31083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:18:56.883241 31084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:18:56.884027 31078 server_base.cc:1034] running on GCE node
I20260924 03:18:56.884660 31078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:18:56.885650 31078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:18:56.886868 31078 hybrid_clock.cc:648] HybridClock initialized: now 1790219936886827 us; error 37 us; skew 500 ppm
I20260924 03:18:56.887149 31078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:18:56 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219936, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:18:56.890372 31078 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:18:56.891788 31078 webserver.cc:460] Webserver started at http://127.14.227.131:42977/ using document root <none> and password file <none>
I20260924 03:18:56.892427 31078 fs_manager.cc:359] Metadata directory not provided
I20260924 03:18:56.892555 31078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:18:56.892848 31078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:18:56.895601 31078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "cef11dfd8e50422fbe937df7e4261a6c"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "ed99e06f3dde8e89df622d7b84c2dd68"
server_key_iv: "f7175b97f319b79841312996f9abf6ae"
server_key_version: "encryptionkey@0"
I20260924 03:18:56.896341 31078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "cef11dfd8e50422fbe937df7e4261a6c"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "ed99e06f3dde8e89df622d7b84c2dd68"
server_key_iv: "f7175b97f319b79841312996f9abf6ae"
server_key_version: "encryptionkey@0"
I20260924 03:18:56.901335 31078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 03:18:56.904937 31093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:56.906282 31078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:18:56.906445 31078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "cef11dfd8e50422fbe937df7e4261a6c"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "ed99e06f3dde8e89df622d7b84c2dd68"
server_key_iv: "f7175b97f319b79841312996f9abf6ae"
server_key_version: "encryptionkey@0"
I20260924 03:18:56.906651 31078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:18:56.934482 31078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:18:56.938670 31078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:18:56.938942 31078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:18:56.939805 31078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:18:56.940996 31078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:18:56.941066 31078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:56.941226 31078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:18:56.941277 31078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:18:56.974078 31078 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:38421
I20260924 03:18:56.974120 31205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:38421 every 8 connection(s)
I20260924 03:18:56.975294 31078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:18:56.977051 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 31078
I20260924 03:18:56.977265 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:18:56.977864 15246 external_mini_cluster.cc:1428] Setting key c7b3ca4517f4a4a3f5480751aee8f742
Sep 24 03:18:56 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219936, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:18:56.991219 30939 heartbeater.cc:503] Master 127.14.227.190:38957 was elected leader, sending a full tablet report...
I20260924 03:18:56.995417 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:56.977896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:43779 (local address 127.14.227.190:38957)
0924 03:18:56.978528 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:56.978536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:56.979538 (+ 1002us) server_negotiation.cc:407] Connection header received
0924 03:18:56.981379 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:56.981384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:56.981477 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:56.981556 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:56.983047 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:56.983895 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:56.985355 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:56.985741 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:56.985831 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:56.988996 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:56.989001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:56.989004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:56.989035 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:56.992207 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:56.993288 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:56.993293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:56.993295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:56.993411 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:56.993812 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:56.993817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:56.993819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:56.994229 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:56.994372 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:56.994981 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 03:18:56.995153 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493}
I20260924 03:18:56.996330 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:56.978587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:43779)
0924 03:18:56.979381 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:56.979429 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:56.981147 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:56.981748 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:56.981756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:56.982365 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:18:56.982901 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:56.982918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:56.984024 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:56.984028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:56.985228 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:56.985239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:56.985918 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:56.985922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:56.986159 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:56.986739 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:18:56.986760 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:18:56.988809 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:18:56.992521 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:56.992527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:56.992538 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:56.993154 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:56.993501 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:18:56.993504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:18:56.993506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:18:56.993662 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:18:56.994357 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:18:56.994362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:18:56.994631 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 03:18:56.994854 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 03:18:56.995202 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":345,"threads_started":1}
I20260924 03:18:56.997735 31206 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38957
I20260924 03:18:56.998059 31206 heartbeater.cc:464] Registering TS with master...
I20260924 03:18:56.998667 31206 heartbeater.cc:511] Master 127.14.227.190:38957 requested a full tablet report, sending...
I20260924 03:18:57.000088 30749 ts_manager.cc:194] Registered new tserver with Master: cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
I20260924 03:18:57.001264 30749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:43779
I20260924 03:18:57.005347 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:18:57.021879 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.009854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.14.227.190:38957)
0924 03:18:57.010150 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.010154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.010640 (+ 486us) server_negotiation.cc:407] Connection header received
0924 03:18:57.010811 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.010815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.010871 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.011007 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:57.012336 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.013067 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.014375 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.014823 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.014919 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.017064 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:57.017069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:57.017072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:57.017100 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:57.019791 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:57.020380 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:57.020384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:57.020386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:57.020512 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:57.020793 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:57.020796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:57.020798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:57.021076 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:57.021261 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.021495 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.021638 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260924 03:18:57.025683 30749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:18:57.032478 30749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:18:57.055445 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.044992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.14.227.129:45663)
0924 03:18:57.047621 (+ 2629us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.047628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.047653 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:18:57.047741 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.047748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.048000 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.048230 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:57.048865 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.050462 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.052652 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.054117 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.054719 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.054874 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.054942 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.055108 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2468,"thread_start_us":2451,"threads_started":1}
I20260924 03:18:57.055495 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.044003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.14.227.130:35029)
0924 03:18:57.046730 (+ 2727us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.046743 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.046778 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:18:57.046880 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.046889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.047135 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.048003 (+ 868us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:57.048220 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.049754 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.052868 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.054191 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.054396 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.054561 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.055005 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.055162 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2503,"thread_start_us":2344,"threads_started":1}
I20260924 03:18:57.055699 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.044372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:41350)
0924 03:18:57.045354 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:57.045371 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:57.045496 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:57.048220 (+ 2724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:57.048229 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:57.048259 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:57.048693 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.048709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.050640 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.050645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.052509 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.052520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.054355 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.054373 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.054561 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.054713 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 03:18:57.054835 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 03:18:57.055270 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":95,"thread_start_us":797,"threads_started":1}
I20260924 03:18:57.055755 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.043686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:35029 (local address 127.0.0.1:44488)
0924 03:18:57.044498 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:57.044544 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:57.044717 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:57.047368 (+ 2651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:57.047375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:57.047403 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:57.047732 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.047745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.049925 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.049931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.052698 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.052706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.054429 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.054434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.054725 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.054754 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:18:57.054835 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:18:57.055044 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":196,"thread_start_us":409,"threads_started":1}
I20260924 03:18:57.056270 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.045860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.14.227.131:38421)
0924 03:18:57.047334 (+ 1474us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.047344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.047378 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:18:57.047474 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.047481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.047695 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.047916 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:57.048622 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.049699 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.052837 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.054170 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.054358 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.054526 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.055771 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.055941 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":1140,"threads_started":1}
I20260924 03:18:57.056372 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.045096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38421 (local address 127.0.0.1:46078)
0924 03:18:57.045998 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:57.046017 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:57.046128 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:57.047906 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:57.047912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:57.047959 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:57.048403 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.048421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.049933 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.049937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.052677 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.052692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.054673 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.054682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.054881 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.054944 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 03:18:57.055032 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:18:57.055146 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":330,"thread_start_us":1070,"threads_started":1}
I20260924 03:18:57.059448 31008 tablet_service.cc:1467] Processing CreateTablet for tablet acc541e4fc70468ca6f47482759cdca9 (DEFAULT_TABLE table=test-table [id=96301255704846a6b207148073f23229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:57.059514 30874 tablet_service.cc:1467] Processing CreateTablet for tablet acc541e4fc70468ca6f47482759cdca9 (DEFAULT_TABLE table=test-table [id=96301255704846a6b207148073f23229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:57.059651 31141 tablet_service.cc:1467] Processing CreateTablet for tablet acc541e4fc70468ca6f47482759cdca9 (DEFAULT_TABLE table=test-table [id=96301255704846a6b207148073f23229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:18:57.060611 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc541e4fc70468ca6f47482759cdca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:57.060820 31141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc541e4fc70468ca6f47482759cdca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:57.060820 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc541e4fc70468ca6f47482759cdca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:18:57.071789 31224 tablet_bootstrap.cc:492] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Bootstrap starting.
I20260924 03:18:57.075484 31225 tablet_bootstrap.cc:492] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Bootstrap starting.
I20260924 03:18:57.075053 31224 tablet_bootstrap.cc:654] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:57.077457 31224 log.cc:824] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:57.078739 31226 tablet_bootstrap.cc:492] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Bootstrap starting.
I20260924 03:18:57.079334 31225 tablet_bootstrap.cc:654] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:57.080827 31225 log.cc:824] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:57.081697 31226 tablet_bootstrap.cc:654] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Neither blocks nor log segments found. Creating new log.
I20260924 03:18:57.083026 31226 log.cc:824] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:18:57.084158 31224 tablet_bootstrap.cc:492] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: No bootstrap required, opened a new log
I20260924 03:18:57.084285 31225 tablet_bootstrap.cc:492] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: No bootstrap required, opened a new log
I20260924 03:18:57.084434 31224 ts_tablet_manager.cc:1397] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260924 03:18:57.084507 31225 ts_tablet_manager.cc:1397] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 03:18:57.086036 31226 tablet_bootstrap.cc:492] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: No bootstrap required, opened a new log
I20260924 03:18:57.086318 31226 ts_tablet_manager.cc:1397] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 03:18:57.093353 31226 raft_consensus.cc:348] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.093731 31226 raft_consensus.cc:374] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:57.093859 31226 raft_consensus.cc:729] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef11dfd8e50422fbe937df7e4261a6c, State: Initialized, Role: FOLLOWER
I20260924 03:18:57.094550 31226 consensus_queue.cc:260] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.094827 31225 raft_consensus.cc:348] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.094827 31224 raft_consensus.cc:348] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.095202 31225 raft_consensus.cc:374] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:57.095199 31224 raft_consensus.cc:374] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:18:57.095285 31225 raft_consensus.cc:729] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87da36cd174d4ec1af9b20b7d0f51271, State: Initialized, Role: FOLLOWER
I20260924 03:18:57.095284 31224 raft_consensus.cc:729] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c603e77e6a14db6883a712ee08493af, State: Initialized, Role: FOLLOWER
I20260924 03:18:57.095907 31225 consensus_queue.cc:260] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.095904 31224 consensus_queue.cc:260] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.096992 31206 heartbeater.cc:503] Master 127.14.227.190:38957 was elected leader, sending a full tablet report...
I20260924 03:18:57.097687 31226 ts_tablet_manager.cc:1428] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 03:18:57.099603 31224 ts_tablet_manager.cc:1428] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260924 03:18:57.100793 31073 heartbeater.cc:503] Master 127.14.227.190:38957 was elected leader, sending a full tablet report...
I20260924 03:18:57.101110 31225 ts_tablet_manager.cc:1428] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
W20260924 03:18:57.211784 30940 tablet.cc:2369] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:18:57.226171 31074 tablet.cc:2369] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:18:57.226879 31207 tablet.cc:2369] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:18:57.230506 31231 raft_consensus.cc:482] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:18:57.230777 31231 raft_consensus.cc:504] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.232450 31231 leader_election.cc:288] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
I20260924 03:18:57.239108 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.232955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:59323 (local address 127.14.227.130:35029)
0924 03:18:57.233137 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.233141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.233543 (+ 402us) server_negotiation.cc:407] Connection header received
0924 03:18:57.233757 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.233763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.233840 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.233965 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:57.234836 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.235701 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.237373 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.238219 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.238377 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.238454 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.238727 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.238849 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:18:57.239142 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.232758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:35029 (local address 127.14.227.129:59323)
0924 03:18:57.233438 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:57.233459 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:57.233603 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:57.234162 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:57.234169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:57.234206 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:57.234653 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.234664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.235829 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.235836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.237223 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.237234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.238430 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.238436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.238576 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.238600 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:18:57.238703 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:18:57.238824 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":342,"threads_started":1}
I20260924 03:18:57.240063 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc541e4fc70468ca6f47482759cdca9" candidate_uuid: "87da36cd174d4ec1af9b20b7d0f51271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c603e77e6a14db6883a712ee08493af" is_pre_election: true
I20260924 03:18:57.240245 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.233744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:51365 (local address 127.14.227.131:38421)
0924 03:18:57.234003 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.234009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.234610 (+ 601us) server_negotiation.cc:407] Connection header received
0924 03:18:57.234806 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.234812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.234884 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.234980 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:18:57.235585 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.236329 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.238352 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.239134 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.239348 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.239420 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.239887 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.240012 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:18:57.240311 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.233590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38421 (local address 127.14.227.129:51365)
0924 03:18:57.234500 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 03:18:57.234528 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 03:18:57.234665 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:18:57.235013 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:18:57.235017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:18:57.235044 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:18:57.235434 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.235445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.236592 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.236605 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.238153 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:18:57.238165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.239545 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.239551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:18:57.239714 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.239738 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:18:57.239839 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:18:57.239953 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":1,"thread_start_us":557,"threads_started":1}
I20260924 03:18:57.240751 31028 raft_consensus.cc:2454] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87da36cd174d4ec1af9b20b7d0f51271 in term 0.
I20260924 03:18:57.241273 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc541e4fc70468ca6f47482759cdca9" candidate_uuid: "87da36cd174d4ec1af9b20b7d0f51271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef11dfd8e50422fbe937df7e4261a6c" is_pre_election: true
I20260924 03:18:57.241466 30827 leader_election.cc:302] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271, 8c603e77e6a14db6883a712ee08493af; no voters:
I20260924 03:18:57.241775 31161 raft_consensus.cc:2454] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87da36cd174d4ec1af9b20b7d0f51271 in term 0.
I20260924 03:18:57.241990 31231 raft_consensus.cc:2781] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:18:57.242162 31231 raft_consensus.cc:482] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:18:57.242331 31231 raft_consensus.cc:3037] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:57.244590 31231 raft_consensus.cc:504] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.245252 31231 leader_election.cc:288] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 1 election: Requested vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
I20260924 03:18:57.245602 31028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc541e4fc70468ca6f47482759cdca9" candidate_uuid: "87da36cd174d4ec1af9b20b7d0f51271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c603e77e6a14db6883a712ee08493af"
I20260924 03:18:57.245697 31161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc541e4fc70468ca6f47482759cdca9" candidate_uuid: "87da36cd174d4ec1af9b20b7d0f51271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef11dfd8e50422fbe937df7e4261a6c"
I20260924 03:18:57.245817 31028 raft_consensus.cc:3037] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:57.245886 31161 raft_consensus.cc:3037] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:18:57.249032 31161 raft_consensus.cc:2454] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87da36cd174d4ec1af9b20b7d0f51271 in term 1.
I20260924 03:18:57.249029 31028 raft_consensus.cc:2454] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87da36cd174d4ec1af9b20b7d0f51271 in term 1.
I20260924 03:18:57.249413 30827 leader_election.cc:302] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271, 8c603e77e6a14db6883a712ee08493af; no voters:
I20260924 03:18:57.249696 31231 raft_consensus.cc:2781] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:18:57.250159 31231 raft_consensus.cc:686] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 LEADER]: Becoming Leader. State: Replica: 87da36cd174d4ec1af9b20b7d0f51271, State: Running, Role: LEADER
I20260924 03:18:57.250658 31231 consensus_queue.cc:237] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } }
I20260924 03:18:57.255232 30749 catalog_manager.cc:5495] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87da36cd174d4ec1af9b20b7d0f51271 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "87da36cd174d4ec1af9b20b7d0f51271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } health_report { overall_health: HEALTHY } } }
I20260924 03:18:57.292999 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.287170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.14.227.129:45663)
0924 03:18:57.287377 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.287382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.287401 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:18:57.287509 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.287513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.287583 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.287729 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:18:57.288114 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.288810 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.290566 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.291061 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.291223 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.291792 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:18:57.292342 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:18:57.292531 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.292598 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.292692 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:18:57.301306 31028 raft_consensus.cc:1261] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 1 FOLLOWER]: Refusing update from remote peer 87da36cd174d4ec1af9b20b7d0f51271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:57.301316 31161 raft_consensus.cc:1261] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 1 FOLLOWER]: Refusing update from remote peer 87da36cd174d4ec1af9b20b7d0f51271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:18:57.302345 31231 consensus_queue.cc:1035] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:57.302776 31235 consensus_queue.cc:1035] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:18:57.315549 31240 mvcc.cc:205] Tried to move back new op lower bound from 7332740863175303168 to 7332740862982045696. Current Snapshot: MvccSnapshot[applied={T|T < 7332740863175303168}]
I20260924 03:18:57.318426 31239 mvcc.cc:205] Tried to move back new op lower bound from 7332740863175303168 to 7332740862982045696. Current Snapshot: MvccSnapshot[applied={T|T < 7332740863175303168}]
I20260924 03:18:57.327330 30748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:18:57.327680 30748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:18:57.331813 30748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Sending DeleteTablet for 3 replicas of tablet acc541e4fc70468ca6f47482759cdca9
I20260924 03:18:57.333060 31141 tablet_service.cc:1514] Processing DeleteTablet for tablet acc541e4fc70468ca6f47482759cdca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:57 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:46078
I20260924 03:18:57.333102 31008 tablet_service.cc:1514] Processing DeleteTablet for tablet acc541e4fc70468ca6f47482759cdca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:57 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:44488
I20260924 03:18:57.334105 31248 tablet_replica.cc:330] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: stopping tablet replica
I20260924 03:18:57.334038 30874 tablet_service.cc:1514] Processing DeleteTablet for tablet acc541e4fc70468ca6f47482759cdca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:18:57 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:41350
I20260924 03:18:57.334664 31248 raft_consensus.cc:2229] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:18:57.335259 31248 raft_consensus.cc:2258] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:57.335813 31247 tablet_replica.cc:330] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: stopping tablet replica
I20260924 03:18:57.336252 31247 raft_consensus.cc:2229] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:18:57.336469 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 30811
I20260924 03:18:57.336848 31247 raft_consensus.cc:2258] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:57.339430 31248 ts_tablet_manager.cc:1905] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:57.347903 31248 ts_tablet_manager.cc:1918] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:57.350013 31248 log.cc:1196] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/acc541e4fc70468ca6f47482759cdca9
I20260924 03:18:57.350111 31247 ts_tablet_manager.cc:1905] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:57.350524 31248 ts_tablet_manager.cc:1939] T acc541e4fc70468ca6f47482759cdca9 P 8c603e77e6a14db6883a712ee08493af: Deleting consensus metadata
I20260924 03:18:57.352242 30734 catalog_manager.cc:4841] TS 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029): tablet acc541e4fc70468ca6f47482759cdca9 (table test-table [id=96301255704846a6b207148073f23229]) successfully deleted
I20260924 03:18:57.357268 31249 tablet_replica.cc:330] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: stopping tablet replica
I20260924 03:18:57.357220 31247 ts_tablet_manager.cc:1918] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:57.357719 31249 raft_consensus.cc:2229] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:18:57.357818 31247 log.cc:1196] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/acc541e4fc70468ca6f47482759cdca9
I20260924 03:18:57.358469 31247 ts_tablet_manager.cc:1939] T acc541e4fc70468ca6f47482759cdca9 P cef11dfd8e50422fbe937df7e4261a6c: Deleting consensus metadata
I20260924 03:18:57.358458 31249 raft_consensus.cc:2258] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:18:57.360217 31249 ts_tablet_manager.cc:1905] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:18:57.360594 30734 catalog_manager.cc:4841] TS cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421): tablet acc541e4fc70468ca6f47482759cdca9 (table test-table [id=96301255704846a6b207148073f23229]) successfully deleted
Sep 24 03:18:57 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:18:57.367376 31249 ts_tablet_manager.cc:1918] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:18:57.367539 31249 log.cc:1196] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/acc541e4fc70468ca6f47482759cdca9
I20260924 03:18:57.368116 31249 ts_tablet_manager.cc:1939] T acc541e4fc70468ca6f47482759cdca9 P 87da36cd174d4ec1af9b20b7d0f51271: Deleting consensus metadata
I20260924 03:18:57.369738 30735 catalog_manager.cc:4841] TS 87da36cd174d4ec1af9b20b7d0f51271 (127.14.227.129:45663): tablet acc541e4fc70468ca6f47482759cdca9 (table test-table [id=96301255704846a6b207148073f23229]) successfully deleted
I20260924 03:18:57.370427 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.337894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.14.227.129:45663)
0924 03:18:57.338093 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.338100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.348740 (+ 10640us) server_negotiation.cc:407] Connection header received
0924 03:18:57.349088 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.349095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.349217 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.349324 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:57.351891 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.353202 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.355494 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.356298 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.356401 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.363035 (+ 6634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:57.363047 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:57.363059 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:57.363094 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:57.366796 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:57.367619 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:57.367632 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:57.367638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:57.367861 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:57.368345 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:57.368357 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:57.368361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:57.368719 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:57.369058 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.369402 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.370062 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:18:57.371057 30934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:57.470310 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 30811
I20260924 03:18:57.488976 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 30945
Sep 24 03:18:57 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:18:57.504556 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.489802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.14.227.130:35029)
0924 03:18:57.490014 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.490018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.490035 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:18:57.490236 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.490240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.490298 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.490379 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:57.491649 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.492364 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.493823 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.494321 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.494409 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.499081 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:57.499104 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:57.499118 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:57.499145 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:57.501758 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:57.502494 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:57.502508 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:57.502510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:57.502623 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:57.503027 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:57.503032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:57.503037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:57.503405 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:57.503706 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.504014 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.504225 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:18:57.505280 31068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:57.594868 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 30945
I20260924 03:18:57.612215 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 31078
Sep 24 03:18:57 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:18:57.626906 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:18:57.612952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.14.227.131:38421)
0924 03:18:57.613183 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:18:57.613188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:18:57.613208 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:18:57.613326 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:18:57.613331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:18:57.613409 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:18:57.613500 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:18:57.614836 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.615563 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.616786 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:18:57.617203 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:18:57.617355 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:18:57.621664 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:18:57.621696 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:18:57.621712 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:18:57.621748 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:18:57.624275 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:57.624877 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:57.624896 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:57.624898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:57.625037 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:18:57.625396 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:18:57.625399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:18:57.625401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:18:57.625725 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:18:57.625926 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 03:18:57.626489 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 03:18:57.626648 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 03:18:57.627563 31201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:57.714900 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 31078
I20260924 03:18:57.730751 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 30718
I20260924 03:18:57.731899 30780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:18:57.841117 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 30718
I20260924 03:19:17.862406 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38957
--webserver_interface=127.14.227.190
--webserver_port=33063
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:19:18.183643 31255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:18.184090 31255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:18.184396 31255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:18.195670 31255 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:19:18.195794 31255 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:19:18.195894 31255 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:19:18.196030 31255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:19:18.196116 31255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:18.196214 31255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:18.196306 31255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:19:18.196386 31255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:19:18.212819 31255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=33063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:18.214351 31255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:18.215647 31255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:18.237821 31261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:18.237772 31262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:18.237802 31264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:18.238231 31255 server_base.cc:1034] running on GCE node
I20260924 03:19:18.239327 31255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:18.240746 31255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:18.242033 31255 hybrid_clock.cc:648] HybridClock initialized: now 1790219958241917 us; error 44 us; skew 500 ppm
I20260924 03:19:18.242393 31255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:18 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:18.245875 31255 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:19:18.247366 31255 webserver.cc:460] Webserver started at http://127.14.227.190:33063/ using document root <none> and password file <none>
I20260924 03:19:18.248095 31255 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:18.248245 31255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:18.254058 31255 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:19:18.256868 31271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:18.258270 31255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:19:18.258458 31255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "2df5d208da25487c88c92b6f54032e20"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "11c92750478807823acc6d510333e378"
server_key_iv: "9fb3d90d06d61d2bed94128385e10fb7"
server_key_version: "encryptionkey@0"
I20260924 03:19:18.259436 31255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:18.275796 31255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:18.279551 31255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:18.279824 31255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:18.298285 31255 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:38957
I20260924 03:19:18.298420 31322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:38957 every 8 connection(s)
I20260924 03:19:18.299880 31255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:19:18.304786 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 31255
I20260924 03:19:18.305580 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:45663
--local_ip_for_outbound_sockets=127.14.227.129
--tserver_master_addrs=127.14.227.190:38957
--webserver_port=41537
--webserver_interface=127.14.227.129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:19:18.307214 31323 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:19:18.311305 31323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Bootstrap starting.
I20260924 03:19:18.330391 31323 log.cc:824] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Log is configured to *not* fsync() on all Append() calls
I20260924 03:19:18.341818 31323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:19:18.342409 31323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Bootstrap complete.
I20260924 03:19:18.351318 31323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:19:18.352147 31323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df5d208da25487c88c92b6f54032e20, State: Initialized, Role: FOLLOWER
I20260924 03:19:18.352778 31323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:19:18.353029 31323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:19:18.353142 31323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:19:18.353317 31323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:19:18.357127 31323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:19:18.357661 31323 leader_election.cc:302] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2df5d208da25487c88c92b6f54032e20; no voters:
I20260924 03:19:18.358340 31323 leader_election.cc:288] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:19:18.358484 31327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:19:18.360767 31323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:19:18.362222 31327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 2 LEADER]: Becoming Leader. State: Replica: 2df5d208da25487c88c92b6f54032e20, State: Running, Role: LEADER
I20260924 03:19:18.363022 31327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:19:18.365657 31329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2df5d208da25487c88c92b6f54032e20. Latest consensus state: current_term: 2 leader_uuid: "2df5d208da25487c88c92b6f54032e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } } }
I20260924 03:19:18.366101 31329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: This master's current role is: LEADER
I20260924 03:19:18.366673 31328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2df5d208da25487c88c92b6f54032e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } } }
I20260924 03:19:18.366989 31328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: This master's current role is: LEADER
I20260924 03:19:18.373251 31338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:19:18.380712 31338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:19:18.382381 31338 catalog_manager.cc:1259] Loaded cluster ID: 6c8b5d4ca75f46848d2bc4a8e9b0b7e4
I20260924 03:19:18.382512 31338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:19:18.387444 31338 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:19:18.389279 31338 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Loaded TSK: 0
I20260924 03:19:18.412412 31338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Generated new TSK 1
I20260924 03:19:18.412889 31338 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:19:18.669691 31325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:18.670020 31325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:18.670203 31325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:18.677433 31325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:18.677542 31325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:18.677719 31325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:19:18.687517 31325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:45663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=41537
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:18.688764 31325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:18.690169 31325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:18.704670 31351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:18.704674 31352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:18.705374 31354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:18.706485 31325 server_base.cc:1034] running on GCE node
I20260924 03:19:18.707351 31325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:18.708662 31325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:18.709926 31325 hybrid_clock.cc:648] HybridClock initialized: now 1790219958709847 us; error 101 us; skew 500 ppm
I20260924 03:19:18.710342 31325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:18 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:18.714352 31325 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:19:18.715881 31325 webserver.cc:460] Webserver started at http://127.14.227.129:41537/ using document root <none> and password file <none>
I20260924 03:19:18.716559 31325 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:18.716711 31325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:18.722402 31325 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 03:19:18.725356 31361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:18.726938 31325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:19:18.727130 31325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "87da36cd174d4ec1af9b20b7d0f51271"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "6b93363eee933cfe654cd9f24c4bbe1b"
server_key_iv: "a97c0d91241a6cb777886adc81899d71"
server_key_version: "encryptionkey@0"
I20260924 03:19:18.728119 31325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:18.751307 31325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:18.755718 31325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:18.755983 31325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:18.757005 31325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:19:18.758466 31325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:19:18.758600 31325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:18.758715 31325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:19:18.758793 31325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:18.793203 31325 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:45663
I20260924 03:19:18.793287 31473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:45663 every 8 connection(s)
I20260924 03:19:18.794714 31325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:19:18.800803 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 31325
I20260924 03:19:18.801595 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:35029
--local_ip_for_outbound_sockets=127.14.227.130
--tserver_master_addrs=127.14.227.190:38957
--webserver_port=37699
--webserver_interface=127.14.227.130
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:19:18 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219958, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:19:18.819916 31477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:18.797539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:49733)
0924 03:19:18.798321 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:18.798370 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:18.800029 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:18.801668 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:18.801675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:18.802298 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:19:18.802814 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:18.802830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:18.805120 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:18.805124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:18.806466 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:18.806478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:18.807997 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:18.808007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:18.808285 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:18.809038 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:19:18.809058 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:19:18.811524 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:19:18.816069 (+ 4545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:18.816075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:18.816085 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:18.816464 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:18.816955 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:18.816958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:18.816960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:18.817121 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:18.818060 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:19:18.818066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:19:18.818360 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 03:19:18.818575 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 03:19:18.818962 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":357,"threads_started":1}
I20260924 03:19:18.820479 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:18.797361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:49733 (local address 127.14.227.190:38957)
0924 03:19:18.797923 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:18.797935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:18.798490 (+ 555us) server_negotiation.cc:407] Connection header received
0924 03:19:18.800633 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:18.800665 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:18.801160 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:18.801587 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:18.803010 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:18.804942 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:18.806615 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:18.807734 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:18.807937 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:18.811800 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:18.811812 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:18.811831 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:18.811879 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:18.815925 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:18.816612 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:18.816623 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:18.816633 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:18.816832 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:18.817235 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:18.817243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:18.817249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:18.817885 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:18.818131 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:18.818845 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 03:19:18.819278 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":259,"threads_started":1}
I20260924 03:19:18.823280 31474 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38957
I20260924 03:19:18.823603 31474 heartbeater.cc:464] Registering TS with master...
I20260924 03:19:18.824295 31474 heartbeater.cc:511] Master 127.14.227.190:38957 requested a full tablet report, sending...
I20260924 03:19:18.826472 31288 ts_manager.cc:194] Registered new tserver with Master: 87da36cd174d4ec1af9b20b7d0f51271 (127.14.227.129:45663)
I20260924 03:19:18.827903 31288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:49733
W20260924 03:19:19.127568 31478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:19.127846 31478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:19.128015 31478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:19.134999 31478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:19.135138 31478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:19.135301 31478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:19:19.144629 31478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:35029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=37699
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:19.145854 31478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:19.147189 31478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:19.159945 31486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:19.160012 31483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:19.160053 31484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:19.161244 31478 server_base.cc:1034] running on GCE node
I20260924 03:19:19.161824 31478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:19.162746 31478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:19.163977 31478 hybrid_clock.cc:648] HybridClock initialized: now 1790219959163929 us; error 51 us; skew 500 ppm
I20260924 03:19:19.164407 31478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:19 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:19.167548 31478 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:19:19.168898 31478 webserver.cc:460] Webserver started at http://127.14.227.130:37699/ using document root <none> and password file <none>
I20260924 03:19:19.169574 31478 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:19.169731 31478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:19.174989 31478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 03:19:19.177606 31493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:19.178978 31478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:19:19.179111 31478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "8c603e77e6a14db6883a712ee08493af"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "42d97d5b2d44b613e983f195ab5e118b"
server_key_iv: "e3dfd539934bba738eeb791b33d4732f"
server_key_version: "encryptionkey@0"
I20260924 03:19:19.180013 31478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:19.203092 31478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:19.207159 31478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:19.207379 31478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:19.208417 31478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:19:19.209712 31478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:19:19.209797 31478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:19.209869 31478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:19:19.209923 31478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:19.241668 31478 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:35029
I20260924 03:19:19.241753 31605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:35029 every 8 connection(s)
I20260924 03:19:19.243206 31478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:19:19.249402 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 31478
I20260924 03:19:19.250126 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:38421
--local_ip_for_outbound_sockets=127.14.227.131
--tserver_master_addrs=127.14.227.190:38957
--webserver_port=42977
--webserver_interface=127.14.227.131
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 03:19:19 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:19:19.263303 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.245793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:52883 (local address 127.14.227.190:38957)
0924 03:19:19.246009 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.246016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.247215 (+ 1199us) server_negotiation.cc:407] Connection header received
0924 03:19:19.248998 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.249005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.249118 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.249280 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:19.250899 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.252065 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.253560 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.254359 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.254510 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.257768 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:19.257773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:19.257776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:19.257805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:19.260254 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:19.260978 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:19.260985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:19.260988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:19.261112 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:19.261532 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:19.261537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:19.261541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:19.261820 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:19.262065 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.262701 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.262936 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:19:19.264287 31608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.246329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:52883)
0924 03:19:19.247023 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:19.247067 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:19.248714 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:19.249449 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:19.249456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:19.250066 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:19:19.250688 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.250718 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.252217 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.252220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.253414 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.253426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.254569 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.254574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.254760 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.255413 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:19:19.255432 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:19:19.257527 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:19:19.260390 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:19.260399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:19.260415 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:19.260873 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:19.261231 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:19.261235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:19.261239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:19.261418 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:19.261969 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:19:19.261977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:19:19.262331 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 03:19:19.262594 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 03:19:19.263094 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
I20260924 03:19:19.265720 31606 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38957
I20260924 03:19:19.266072 31606 heartbeater.cc:464] Registering TS with master...
I20260924 03:19:19.266763 31606 heartbeater.cc:511] Master 127.14.227.190:38957 requested a full tablet report, sending...
I20260924 03:19:19.268214 31288 ts_manager.cc:194] Registered new tserver with Master: 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029)
I20260924 03:19:19.269013 31288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:52883
W20260924 03:19:19.574798 31609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:19.575115 31609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:19.575273 31609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:19.582429 31609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:19.582569 31609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:19.582738 31609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:19:19.592357 31609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:38421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=42977
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:19.593591 31609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:19.594864 31609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:19.607621 31614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:19.607584 31617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:19.607682 31615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:19.608498 31609 server_base.cc:1034] running on GCE node
I20260924 03:19:19.609076 31609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:19.610095 31609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:19.611286 31609 hybrid_clock.cc:648] HybridClock initialized: now 1790219959611233 us; error 51 us; skew 500 ppm
I20260924 03:19:19.611584 31609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:19 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:19.614719 31609 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:19:19.616039 31609 webserver.cc:460] Webserver started at http://127.14.227.131:42977/ using document root <none> and password file <none>
I20260924 03:19:19.616696 31609 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:19.616845 31609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:19.621860 31609 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:19:19.624439 31624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:19.625734 31609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:19:19.625906 31609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "cef11dfd8e50422fbe937df7e4261a6c"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "ed99e06f3dde8e89df622d7b84c2dd68"
server_key_iv: "f7175b97f319b79841312996f9abf6ae"
server_key_version: "encryptionkey@0"
I20260924 03:19:19.626922 31609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:19.686532 31609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:19.690459 31609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:19.690724 31609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:19.691710 31609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:19:19.693013 31609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:19:19.693121 31609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:19.693240 31609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:19:19.693328 31609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:19.726491 31609 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:38421
I20260924 03:19:19.726578 31736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:38421 every 8 connection(s)
I20260924 03:19:19.727943 31609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:19:19.735760 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 31609
Sep 24 03:19:19 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219959, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:19:19.748654 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.730546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:43547 (local address 127.14.227.190:38957)
0924 03:19:19.730729 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.730734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.731952 (+ 1218us) server_negotiation.cc:407] Connection header received
0924 03:19:19.734038 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.734044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.734118 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.734211 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:19.735644 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.736682 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.738523 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.739266 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.739400 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.742768 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:19.742772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:19.742776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:19.742808 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:19.745526 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:19.746349 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:19.746355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:19.746358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:19.746493 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:19.746979 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:19.746985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:19.746988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:19.747423 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:19.747578 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.748193 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.748391 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:19:19.749745 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.731029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:43547)
0924 03:19:19.731763 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:19.731805 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:19.733659 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:19.734410 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:19.734418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:19.734925 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:19:19.735470 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.735486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.736849 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.736857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.738383 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.738395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.739466 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.739470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.739654 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.740312 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:19:19.740332 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:19:19.742556 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:19:19.745673 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:19.745682 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:19.745704 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:19.746247 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:19.746611 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:19.746617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:19.746620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:19.746841 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:19.747547 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:19:19.747553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:19:19.747846 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 03:19:19.748049 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 03:19:19.748425 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":309,"threads_started":1}
I20260924 03:19:19.750949 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.736718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.14.227.190:38957)
0924 03:19:19.737255 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.737261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.737523 (+ 262us) server_negotiation.cc:407] Connection header received
0924 03:19:19.737719 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.737731 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.737782 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.737893 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:19.739884 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.740708 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.742507 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.742934 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.743131 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.745519 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:19.745524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:19.745528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:19.745555 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:19.748311 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:19.749050 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:19.749053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:19.749055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:19.749153 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:19.749541 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:19.749544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:19.749546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:19.749755 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:19.750055 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.750365 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.750568 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":238,"threads_started":1}
I20260924 03:19:19.751250 31737 heartbeater.cc:347] Connected to a master server at 127.14.227.190:38957
I20260924 03:19:19.751596 31737 heartbeater.cc:464] Registering TS with master...
I20260924 03:19:19.752350 31737 heartbeater.cc:511] Master 127.14.227.190:38957 requested a full tablet report, sending...
I20260924 03:19:19.753641 31288 ts_manager.cc:194] Registered new tserver with Master: cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
I20260924 03:19:19.754670 31288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:43547
I20260924 03:19:19.765388 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:19:19.771481 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.766309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.14.227.190:38957)
0924 03:19:19.766517 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.766526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.766872 (+ 346us) server_negotiation.cc:407] Connection header received
0924 03:19:19.767021 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.767025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.767146 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.767261 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:19:19.767656 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.768344 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.769740 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.770219 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.770310 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.770770 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:19:19.770952 (+ 182us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 03:19:19.771207 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52456: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 03:19:19.771605 31740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52456: token expired
W20260924 03:19:19.772113 31210 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.227.190:38957)
I20260924 03:19:19.783972 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.772508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.14.227.190:38957)
0924 03:19:19.772657 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.772661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.772739 (+ 78us) server_negotiation.cc:407] Connection header received
0924 03:19:19.772843 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.772847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.772894 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.772965 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:19.774285 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.774927 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.776249 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.776649 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.776745 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.779301 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:19.779306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:19.779309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:19.779336 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:19.781700 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:19.782341 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:19.782345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:19.782347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:19.782468 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:19.782784 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:19.782787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:19.782789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:19.783079 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:19.783331 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.783586 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.783730 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:19:19.787173 31288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:19:19.795718 31288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:19:19.818624 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.807167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.14.227.130:35029)
0924 03:19:19.807921 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.807931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.809363 (+ 1432us) server_negotiation.cc:407] Connection header received
0924 03:19:19.809450 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.809456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.809643 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.809811 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:19:19.811009 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.812545 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.815860 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.817366 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.817518 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.817682 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.818072 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.818270 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":433,"threads_started":1}
I20260924 03:19:19.818881 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.807618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.14.227.129:45663)
0924 03:19:19.808192 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.808210 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.809106 (+ 896us) server_negotiation.cc:407] Connection header received
0924 03:19:19.809414 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.809421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.809643 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.809798 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:19:19.810866 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.812510 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.815860 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.817366 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.817538 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.817682 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.818394 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.818559 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":275,"threads_started":1}
I20260924 03:19:19.819024 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.808143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:44592)
0924 03:19:19.808957 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:19.809011 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:19.809182 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:19.810097 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:19.810100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:19.810172 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:19:19.810659 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.810675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.812690 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.812695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.815676 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.815691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.817907 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.817913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.818140 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.818232 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 03:19:19.818347 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 03:19:19.818567 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":140,"spinlock_wait_cycles":12160,"thread_start_us":559,"threads_started":1}
I20260924 03:19:19.819669 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.807102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:35029 (local address 127.0.0.1:38390)
0924 03:19:19.808957 (+ 1855us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:19.809011 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:19.809182 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:19.810081 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:19.810091 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:19.810159 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:19:19.810699 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.810717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.812690 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.812710 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.815684 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.815691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.817557 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.817563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.817762 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.817830 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 03:19:19.818820 (+ 990us) client_negotiation.cc:240] Negotiation successful
0924 03:19:19.818968 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"mutex_wait_us":203,"spinlock_wait_cycles":2816,"thread_start_us":1401,"threads_started":1}
I20260924 03:19:19.823012 31409 tablet_service.cc:1467] Processing CreateTablet for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd (DEFAULT_TABLE table=test-table [id=d07950aedfcc490cbfa9822e35b6a4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:19:19.823014 31541 tablet_service.cc:1467] Processing CreateTablet for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd (DEFAULT_TABLE table=test-table [id=d07950aedfcc490cbfa9822e35b6a4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:19:19.824453 31541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:19:19.824469 31409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:19:19.824836 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.809871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.14.227.131:38421)
0924 03:19:19.810595 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.810604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.817289 (+ 6685us) server_negotiation.cc:407] Connection header received
0924 03:19:19.817450 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.817460 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.817726 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.817923 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:19:19.818585 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.821062 (+ 2477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.822784 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.823766 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.823977 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.824149 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.824329 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.824491 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":386,"threads_started":1}
I20260924 03:19:19.825670 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.809299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38421 (local address 127.0.0.1:58092)
0924 03:19:19.817167 (+ 7868us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:19.817181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:19.817317 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:19.817918 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:19.817921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:19.817968 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:19:19.818394 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.818405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.821207 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.821214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.822630 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.822642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.823968 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.823974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.824134 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.824168 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:19:19.824278 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 03:19:19.824396 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7752,"mutex_wait_us":22,"thread_start_us":7640,"threads_started":1}
I20260924 03:19:19.828585 31672 tablet_service.cc:1467] Processing CreateTablet for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd (DEFAULT_TABLE table=test-table [id=d07950aedfcc490cbfa9822e35b6a4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:19:19.829731 31672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:19:19.830842 31474 heartbeater.cc:503] Master 127.14.227.190:38957 was elected leader, sending a full tablet report...
I20260924 03:19:19.839871 31753 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Bootstrap starting.
I20260924 03:19:19.840610 31754 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Bootstrap starting.
I20260924 03:19:19.843472 31755 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Bootstrap starting.
I20260924 03:19:19.844484 31754 tablet_bootstrap.cc:654] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Neither blocks nor log segments found. Creating new log.
I20260924 03:19:19.845155 31753 tablet_bootstrap.cc:654] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Neither blocks nor log segments found. Creating new log.
I20260924 03:19:19.845510 31754 log.cc:824] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Log is configured to *not* fsync() on all Append() calls
I20260924 03:19:19.846345 31753 log.cc:824] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Log is configured to *not* fsync() on all Append() calls
I20260924 03:19:19.846781 31755 tablet_bootstrap.cc:654] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Neither blocks nor log segments found. Creating new log.
I20260924 03:19:19.848228 31755 log.cc:824] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:19:19.848500 31754 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: No bootstrap required, opened a new log
I20260924 03:19:19.848747 31754 ts_tablet_manager.cc:1397] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260924 03:19:19.851257 31755 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: No bootstrap required, opened a new log
I20260924 03:19:19.851501 31755 ts_tablet_manager.cc:1397] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 03:19:19.852401 31753 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: No bootstrap required, opened a new log
I20260924 03:19:19.852649 31753 ts_tablet_manager.cc:1397] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260924 03:19:19.856433 31754 raft_consensus.cc:348] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:19.856835 31754 raft_consensus.cc:374] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:19:19.856950 31754 raft_consensus.cc:729] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87da36cd174d4ec1af9b20b7d0f51271, State: Initialized, Role: FOLLOWER
I20260924 03:19:19.858006 31754 consensus_queue.cc:260] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:19.859652 31754 ts_tablet_manager.cc:1428] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260924 03:19:19.862134 31755 raft_consensus.cc:348] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:19.862516 31755 raft_consensus.cc:374] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:19:19.862618 31755 raft_consensus.cc:729] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef11dfd8e50422fbe937df7e4261a6c, State: Initialized, Role: FOLLOWER
I20260924 03:19:19.862928 31753 raft_consensus.cc:348] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:19.863324 31753 raft_consensus.cc:374] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:19:19.863184 31755 consensus_queue.cc:260] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:19.863405 31753 raft_consensus.cc:729] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c603e77e6a14db6883a712ee08493af, State: Initialized, Role: FOLLOWER
I20260924 03:19:19.864158 31753 consensus_queue.cc:260] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:19.865351 31606 heartbeater.cc:503] Master 127.14.227.190:38957 was elected leader, sending a full tablet report...
I20260924 03:19:19.866273 31753 ts_tablet_manager.cc:1428] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Time spent starting tablet: real 0.013s user 0.006s sys 0.009s
I20260924 03:19:19.866677 31737 heartbeater.cc:503] Master 127.14.227.190:38957 was elected leader, sending a full tablet report...
I20260924 03:19:19.867192 31755 ts_tablet_manager.cc:1428] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260924 03:19:19.959506 31759 raft_consensus.cc:482] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:19:19.959861 31759 raft_consensus.cc:504] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:19.961581 31759 leader_election.cc:288] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
I20260924 03:19:19.968452 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.962092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:32773 (local address 127.14.227.130:35029)
0924 03:19:19.962341 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.962348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.962858 (+ 510us) server_negotiation.cc:407] Connection header received
0924 03:19:19.963086 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.963091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.963168 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.963270 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:19:19.964100 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.964975 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.966797 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.967540 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.967728 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.967802 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.968065 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.968189 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:19:19.968511 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.961947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:35029 (local address 127.14.227.129:32773)
0924 03:19:19.962748 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:19.962772 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:19.962919 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:19.963497 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:19.963501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:19.963524 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:19:19.963954 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.963964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.965105 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.965112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.966638 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.966649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.967740 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.967746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.967910 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.967934 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:19:19.968040 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 03:19:19.968161 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":379,"threads_started":1}
I20260924 03:19:19.969333 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.963157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:57991 (local address 127.14.227.131:38421)
0924 03:19:19.963363 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:19.963369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:19.963772 (+ 403us) server_negotiation.cc:407] Connection header received
0924 03:19:19.963975 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:19.963981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:19.964064 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:19.964173 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:19:19.964807 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.965550 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.967414 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.968240 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.968436 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.968527 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:19.968938 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 03:19:19.969072 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 03:19:19.969431 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:19.962887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38421 (local address 127.14.227.129:57991)
0924 03:19:19.963597 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:19.963662 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:19.963832 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:19.964188 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:19.964193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:19.964216 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:19:19.964567 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.964576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.965699 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.965705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.967250 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:19.967262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:19.968508 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:19.968515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:19.968759 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:19.968798 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:19:19.968899 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:19:19.969023 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":11,"thread_start_us":398,"threads_started":1}
I20260924 03:19:19.969595 31561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702b4a0509ac4c4b94037ce3bfe5b9bd" candidate_uuid: "87da36cd174d4ec1af9b20b7d0f51271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c603e77e6a14db6883a712ee08493af" is_pre_election: true
I20260924 03:19:19.970201 31561 raft_consensus.cc:2454] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87da36cd174d4ec1af9b20b7d0f51271 in term 0.
I20260924 03:19:19.970511 31692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702b4a0509ac4c4b94037ce3bfe5b9bd" candidate_uuid: "87da36cd174d4ec1af9b20b7d0f51271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef11dfd8e50422fbe937df7e4261a6c" is_pre_election: true
I20260924 03:19:19.970956 31362 leader_election.cc:302] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271, 8c603e77e6a14db6883a712ee08493af; no voters:
I20260924 03:19:19.971004 31692 raft_consensus.cc:2454] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87da36cd174d4ec1af9b20b7d0f51271 in term 0.
I20260924 03:19:19.971356 31759 raft_consensus.cc:2781] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:19:19.971518 31759 raft_consensus.cc:482] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:19:19.971630 31759 raft_consensus.cc:3037] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:19:19.973747 31759 raft_consensus.cc:504] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:19.974321 31759 leader_election.cc:288] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 1 election: Requested vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
I20260924 03:19:19.974790 31561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702b4a0509ac4c4b94037ce3bfe5b9bd" candidate_uuid: "87da36cd174d4ec1af9b20b7d0f51271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c603e77e6a14db6883a712ee08493af"
I20260924 03:19:19.974859 31692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702b4a0509ac4c4b94037ce3bfe5b9bd" candidate_uuid: "87da36cd174d4ec1af9b20b7d0f51271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef11dfd8e50422fbe937df7e4261a6c"
I20260924 03:19:19.974967 31561 raft_consensus.cc:3037] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:19:19.975122 31692 raft_consensus.cc:3037] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:19:19.977186 31561 raft_consensus.cc:2454] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87da36cd174d4ec1af9b20b7d0f51271 in term 1.
I20260924 03:19:19.977327 31692 raft_consensus.cc:2454] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87da36cd174d4ec1af9b20b7d0f51271 in term 1.
I20260924 03:19:19.977627 31362 leader_election.cc:302] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271, 8c603e77e6a14db6883a712ee08493af; no voters:
I20260924 03:19:19.977953 31759 raft_consensus.cc:2781] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:19:19.978427 31759 raft_consensus.cc:686] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 LEADER]: Becoming Leader. State: Replica: 87da36cd174d4ec1af9b20b7d0f51271, State: Running, Role: LEADER
I20260924 03:19:19.978893 31759 consensus_queue.cc:237] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
W20260924 03:19:19.979534 31738 tablet.cc:2369] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:19:19.983314 31287 catalog_manager.cc:5495] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87da36cd174d4ec1af9b20b7d0f51271 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "87da36cd174d4ec1af9b20b7d0f51271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } health_report { overall_health: UNKNOWN } } }
W20260924 03:19:19.995163 31607 tablet.cc:2369] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:19:20.007736 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.002846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.14.227.129:45663)
0924 03:19:20.003030 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:20.003034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:20.003051 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:19:20.003219 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:20.003223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:20.003297 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:20.003434 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:19:20.003711 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:20.004408 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:20.005675 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:20.006139 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:20.006229 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:20.006642 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:19:20.007213 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:19:20.007324 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:20.007394 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 03:19:20.007452 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:19:20.015724 31561 raft_consensus.cc:1261] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 1 FOLLOWER]: Refusing update from remote peer 87da36cd174d4ec1af9b20b7d0f51271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:19:20.015723 31692 raft_consensus.cc:1261] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 1 FOLLOWER]: Refusing update from remote peer 87da36cd174d4ec1af9b20b7d0f51271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:19:20.016690 31764 consensus_queue.cc:1035] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:19:20.017050 31759 consensus_queue.cc:1035] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:19:20.042070 31288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:19:20.042325 31288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:19:20.045591 31288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Sending DeleteTablet for 3 replicas of tablet 702b4a0509ac4c4b94037ce3bfe5b9bd
I20260924 03:19:20.046814 31541 tablet_service.cc:1514] Processing DeleteTablet for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:19:20 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38390
I20260924 03:19:20.046820 31409 tablet_service.cc:1514] Processing DeleteTablet for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:19:20 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:44592
I20260924 03:19:20.047217 31672 tablet_service.cc:1514] Processing DeleteTablet for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:19:20 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:58092
I20260924 03:19:20.047878 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 31325
I20260924 03:19:20.058495 31778 tablet_replica.cc:330] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: stopping tablet replica
I20260924 03:19:20.059019 31778 raft_consensus.cc:2229] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:19:20.059631 31778 raft_consensus.cc:2258] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:19:20.066610 31778 ts_tablet_manager.cc:1905] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:19:20.074424 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.049574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.14.227.129:45663)
0924 03:19:20.049840 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:20.049849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:20.049874 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:19:20.049940 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:20.049947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:20.050115 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:20.050244 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:20.052915 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:20.054619 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:20.057204 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:20.058188 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:20.058362 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:20.063490 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:20.063579 (+ 89us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:20.063602 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:20.063676 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:20.068975 (+ 5299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:20.070164 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:20.070177 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:20.070184 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:20.070433 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:20.070984 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:20.070992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:20.070998 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:20.071579 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:20.071921 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:20.073465 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0924 03:19:20.073786 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 03:19:20.075568 31469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:19:20.082161 31778 ts_tablet_manager.cc:1918] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:19:20.082388 31778 log.cc:1196] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/702b4a0509ac4c4b94037ce3bfe5b9bd
I20260924 03:19:20.084818 31778 ts_tablet_manager.cc:1939] T 702b4a0509ac4c4b94037ce3bfe5b9bd P cef11dfd8e50422fbe937df7e4261a6c: Deleting consensus metadata
I20260924 03:19:20.084846 31777 tablet_replica.cc:330] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: stopping tablet replica
I20260924 03:19:20.085911 31777 raft_consensus.cc:2229] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:19:20.086817 31272 catalog_manager.cc:4841] TS cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421): tablet 702b4a0509ac4c4b94037ce3bfe5b9bd (table test-table [id=d07950aedfcc490cbfa9822e35b6a4b9]) successfully deleted
I20260924 03:19:20.087366 31777 raft_consensus.cc:2258] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:19:20.089542 31777 ts_tablet_manager.cc:1905] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:19:20.096722 31777 ts_tablet_manager.cc:1918] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:19:20.096907 31777 log.cc:1196] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/702b4a0509ac4c4b94037ce3bfe5b9bd
I20260924 03:19:20.097431 31777 ts_tablet_manager.cc:1939] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 8c603e77e6a14db6883a712ee08493af: Deleting consensus metadata
I20260924 03:19:20.099263 31272 catalog_manager.cc:4841] TS 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029): tablet 702b4a0509ac4c4b94037ce3bfe5b9bd (table test-table [id=d07950aedfcc490cbfa9822e35b6a4b9]) successfully deleted
I20260924 03:19:20.182322 31776 tablet_replica.cc:330] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: stopping tablet replica
I20260924 03:19:20.182626 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 31325
W20260924 03:19:20.200354 31273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.227.129:45663 (error 108)
I20260924 03:19:20.200881 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 31478
I20260924 03:19:20.202327 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.201852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:44638)
0924 03:19:20.202049 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:20.202164 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:19:20.202589 31273 catalog_manager.cc:4584] TS 87da36cd174d4ec1af9b20b7d0f51271 (127.14.227.129:45663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 702b4a0509ac4c4b94037ce3bfe5b9bd: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45663: connect: Connection refused (error 111)
I20260924 03:19:20.217622 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.201780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.14.227.130:35029)
0924 03:19:20.201980 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:20.201986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:20.202006 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:19:20.202218 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:20.202223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:20.202292 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:20.202371 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:20.203813 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:20.204523 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:20.205739 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:20.206193 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:20.206294 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:20.212490 (+ 6196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:20.212504 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:20.212518 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:20.212552 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:20.215338 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:20.215932 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:20.215952 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:20.215954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:20.216058 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:20.216389 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:20.216392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:20.216394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:20.216739 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:20.216962 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:20.217196 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 03:19:20.217374 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 03:19:20.218379 31601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:19:20.256265 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.255698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:44650)
0924 03:19:20.255931 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:20.256072 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 03:19:20.257290 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.256917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:44664)
0924 03:19:20.257071 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:20.257154 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 03:19:20.306715 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 31478
I20260924 03:19:20.317839 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.317225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:44666)
0924 03:19:20.317498 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:20.317639 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 03:19:20.318935 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.318500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:44674)
0924 03:19:20.318684 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:20.318791 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 03:19:20.325254 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 31609
I20260924 03:19:20.337890 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.326075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.14.227.131:38421)
0924 03:19:20.326239 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:20.326243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:20.326271 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:19:20.326392 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:20.326396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:20.326460 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:20.326542 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:20.327810 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:20.328523 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:20.329714 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:20.330269 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:20.330382 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:20.332814 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:20.332828 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:20.332843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:20.332879 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:20.335490 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:20.336135 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:20.336142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:20.336144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:20.336267 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:20.336563 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:20.336566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:20.336568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:20.336899 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:20.337113 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:20.337365 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 03:19:20.337513 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:19:20.338557 31732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:19:20.399490 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.398912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:44688)
0924 03:19:20.399197 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:20.399309 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 03:19:20.400574 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:20.400168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45663 (local address 127.0.0.1:44696)
0924 03:19:20.400372 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:20.400452 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 03:19:20.427367 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 31609
I20260924 03:19:20.444307 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 31255
I20260924 03:19:20.445421 31318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:19:20.560467 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 31255
I20260924 03:19:40.582208 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38957
--webserver_interface=127.14.227.190
--webserver_port=33063
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:19:40.915022 31784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:40.915287 31784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:40.915488 31784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:40.922638 31784 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:19:40.922727 31784 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:19:40.922806 31784 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:19:40.922907 31784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:19:40.922998 31784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:40.923122 31784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:40.923185 31784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:19:40.923285 31784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:19:40.932799 31784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=33063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:40.934042 31784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:40.935361 31784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:40.947108 31792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:40.947242 31790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:40.947511 31789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:40.948207 31784 server_base.cc:1034] running on GCE node
I20260924 03:19:40.948978 31784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:40.950441 31784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:40.951745 31784 hybrid_clock.cc:648] HybridClock initialized: now 1790219980951678 us; error 33 us; skew 500 ppm
I20260924 03:19:40.952143 31784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:40 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219980, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:40.955590 31784 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:19:40.956933 31784 webserver.cc:460] Webserver started at http://127.14.227.190:33063/ using document root <none> and password file <none>
I20260924 03:19:40.957612 31784 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:40.957747 31784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:40.963083 31784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:19:40.965785 31799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:40.967060 31784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:19:40.967240 31784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "2df5d208da25487c88c92b6f54032e20"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "11c92750478807823acc6d510333e378"
server_key_iv: "9fb3d90d06d61d2bed94128385e10fb7"
server_key_version: "encryptionkey@0"
I20260924 03:19:40.968163 31784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:40.994004 31784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:40.997913 31784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:40.998231 31784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:41.017133 31784 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:38957
I20260924 03:19:41.017172 31850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:38957 every 8 connection(s)
I20260924 03:19:41.018724 31784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:19:41.025391 31851 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:19:41.026187 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 31784
I20260924 03:19:41.027204 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:45663
--local_ip_for_outbound_sockets=127.14.227.129
--tserver_master_addrs=127.14.227.190:38957
--webserver_port=41537
--webserver_interface=127.14.227.129
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:19:41.030328 31851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Bootstrap starting.
I20260924 03:19:41.052119 31851 log.cc:824] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Log is configured to *not* fsync() on all Append() calls
I20260924 03:19:41.067757 31851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:19:41.068296 31851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Bootstrap complete.
I20260924 03:19:41.077742 31851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:19:41.078670 31851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df5d208da25487c88c92b6f54032e20, State: Initialized, Role: FOLLOWER
I20260924 03:19:41.079288 31851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:19:41.079524 31851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:19:41.079649 31851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:19:41.079813 31851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:19:41.083513 31851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:19:41.083963 31851 leader_election.cc:302] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2df5d208da25487c88c92b6f54032e20; no voters:
I20260924 03:19:41.084618 31851 leader_election.cc:288] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:19:41.084753 31855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:19:41.086092 31855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [term 3 LEADER]: Becoming Leader. State: Replica: 2df5d208da25487c88c92b6f54032e20, State: Running, Role: LEADER
I20260924 03:19:41.086655 31855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } }
I20260924 03:19:41.086848 31851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:19:41.089061 31857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2df5d208da25487c88c92b6f54032e20. Latest consensus state: current_term: 3 leader_uuid: "2df5d208da25487c88c92b6f54032e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } } }
I20260924 03:19:41.089433 31857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: This master's current role is: LEADER
I20260924 03:19:41.089222 31856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2df5d208da25487c88c92b6f54032e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df5d208da25487c88c92b6f54032e20" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 38957 } } }
I20260924 03:19:41.089664 31856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20 [sys.catalog]: This master's current role is: LEADER
I20260924 03:19:41.090395 31862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:19:41.107225 31862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:19:41.109616 31862 catalog_manager.cc:1259] Loaded cluster ID: 6c8b5d4ca75f46848d2bc4a8e9b0b7e4
I20260924 03:19:41.109863 31862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:19:41.116582 31862 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:19:41.119066 31862 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Loaded TSK: 0
I20260924 03:19:41.119350 31862 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2df5d208da25487c88c92b6f54032e20: Loaded TSK: 1
F20260924 03:19:41.121024 31862 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f98731689ad google::LogMessage::Fail() at ??:0
@ 0x7f987316b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f98731684de google::LogMessage::Flush() at ??:0
@ 0x7f9873169a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9876ecc7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9876ebb73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9888b971c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9888b96783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9888c222a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9888c220c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9888c6c5e9 std::function<>::operator()() at ??:0
@ 0x7f9888b9acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9888b98c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9888c20610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9888c20439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9875db4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9875db791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9875db7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9875d92eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f987906c6db start_thread at ??:0
@ 0x7f987200261f clone at ??:0
W20260924 03:19:41.420359 31853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:41.420697 31853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:41.420885 31853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:41.428263 31853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:41.428416 31853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:41.428625 31853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:19:41.438602 31853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:45663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=41537
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:41.439884 31853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:41.441119 31853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:41.455381 31898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:41.455425 31901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:41.455268 31899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:41.456276 31853 server_base.cc:1034] running on GCE node
I20260924 03:19:41.456959 31853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:41.457995 31853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:41.459245 31853 hybrid_clock.cc:648] HybridClock initialized: now 1790219981459185 us; error 44 us; skew 500 ppm
I20260924 03:19:41.459576 31853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:41 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219981, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:41.462723 31853 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:19:41.464246 31853 webserver.cc:460] Webserver started at http://127.14.227.129:41537/ using document root <none> and password file <none>
I20260924 03:19:41.464967 31853 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:41.465118 31853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:41.470535 31853 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:19:41.473484 31908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:41.474830 31853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:19:41.475023 31853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "87da36cd174d4ec1af9b20b7d0f51271"
format_stamp: "Formatted at 2026-09-24 03:18:55 on dist-test-slave-vd0p"
server_key: "6b93363eee933cfe654cd9f24c4bbe1b"
server_key_iv: "a97c0d91241a6cb777886adc81899d71"
server_key_version: "encryptionkey@0"
I20260924 03:19:41.475967 31853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:41.495729 31853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:41.499686 31853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:41.500000 31853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:41.501005 31853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:19:41.502894 31915 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 03:19:41.508589 31853 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:19:41.508745 31853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 03:19:41.508916 31853 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:19:41.512116 31853 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:19:41.512223 31853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260924 03:19:41.512552 31915 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Bootstrap starting.
I20260924 03:19:41.537283 31915 log.cc:824] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Log is configured to *not* fsync() on all Append() calls
I20260924 03:19:41.555330 31915 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:19:41.556169 31915 tablet_bootstrap.cc:492] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Bootstrap complete.
I20260924 03:19:41.558284 31915 ts_tablet_manager.cc:1397] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.031s
I20260924 03:19:41.560274 31853 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:45663
I20260924 03:19:41.560320 32022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:45663 every 8 connection(s)
I20260924 03:19:41.561872 31853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:19:41.563753 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 31853
I20260924 03:19:41.564379 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:35029
--local_ip_for_outbound_sockets=127.14.227.130
--tserver_master_addrs=127.14.227.190:38957
--webserver_port=37699
--webserver_interface=127.14.227.130
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:19:41.566877 31915 raft_consensus.cc:348] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:41.568192 31915 raft_consensus.cc:729] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87da36cd174d4ec1af9b20b7d0f51271, State: Initialized, Role: FOLLOWER
I20260924 03:19:41.569124 31915 consensus_queue.cc:260] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:41.570464 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:41.567839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:45617)
0924 03:19:41.568874 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:41.569212 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":449,"threads_started":1}
I20260924 03:19:41.570835 31915 ts_tablet_manager.cc:1428] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260924 03:19:41.571151 32023 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:38957 (0 consecutive failures): Network error: Failed to ping master at 127.14.227.190:38957: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
I20260924 03:19:41.572069 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:41.571738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:38215)
0924 03:19:41.571891 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:41.571967 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 03:19:41.573112 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:41.572698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:60375)
0924 03:19:41.572853 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:41.572954 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:19:41.573390 32023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:19:41.890177 32027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:41.890429 32027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:41.890614 32027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:41.897719 32027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:41.897812 32027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:41.898012 32027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:19:41.907486 32027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:35029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=37699
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:41.908803 32027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:41.910135 32027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:41.923141 32032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:41.923244 32035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:41.923265 32033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:41.924335 32027 server_base.cc:1034] running on GCE node
I20260924 03:19:41.924953 32027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:41.925899 32027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:41.927170 32027 hybrid_clock.cc:648] HybridClock initialized: now 1790219981927123 us; error 40 us; skew 500 ppm
I20260924 03:19:41.927479 32027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:41 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219981, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:41.930454 32027 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:19:41.931803 32027 webserver.cc:460] Webserver started at http://127.14.227.130:37699/ using document root <none> and password file <none>
I20260924 03:19:41.932500 32027 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:41.932646 32027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:41.937676 32027 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:19:41.940373 32042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:41.941686 32027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:19:41.941859 32027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "8c603e77e6a14db6883a712ee08493af"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "42d97d5b2d44b613e983f195ab5e118b"
server_key_iv: "e3dfd539934bba738eeb791b33d4732f"
server_key_version: "encryptionkey@0"
I20260924 03:19:41.942889 32027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:41.958626 32027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:41.962406 32027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:41.962687 32027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:41.963626 32027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:19:41.964911 32027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:19:41.965050 32027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:41.965183 32027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:19:41.965273 32027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:41.999073 32027 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:35029
I20260924 03:19:41.999130 32154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:35029 every 8 connection(s)
I20260924 03:19:42.000527 32027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:19:42.005235 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:42.003519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:54607)
0924 03:19:42.004212 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:42.004434 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":267,"threads_started":1}
W20260924 03:19:42.005836 32155 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:38957 (0 consecutive failures): Network error: Failed to ping master at 127.14.227.190:38957: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
I20260924 03:19:42.006632 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:42.006317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:58671)
0924 03:19:42.006465 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:42.006535 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 03:19:42.007473 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:42.007170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:57655)
0924 03:19:42.007295 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:42.007373 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 03:19:42.007697 32155 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:19:42.008759 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 32027
I20260924 03:19:42.009322 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:38421
--local_ip_for_outbound_sockets=127.14.227.131
--tserver_master_addrs=127.14.227.190:38957
--webserver_port=42977
--webserver_interface=127.14.227.131
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:19:42.328039 32158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:42.328310 32158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:42.328511 32158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:42.335669 32158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:42.335794 32158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:42.335979 32158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:19:42.345345 32158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:38421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=42977
--tserver_master_addrs=127.14.227.190:38957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:42.346551 32158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:42.347790 32158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:42.360853 32166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:42.360934 32164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:42.360898 32163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:42.361874 32158 server_base.cc:1034] running on GCE node
I20260924 03:19:42.362478 32158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:42.363412 32158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:42.364636 32158 hybrid_clock.cc:648] HybridClock initialized: now 1790219982364598 us; error 51 us; skew 500 ppm
I20260924 03:19:42.364940 32158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:42 dist-test-slave-vd0p krb5kdc[30705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219982, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:42.367911 32158 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:19:42.369333 32158 webserver.cc:460] Webserver started at http://127.14.227.131:42977/ using document root <none> and password file <none>
I20260924 03:19:42.370028 32158 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:42.370175 32158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:42.375643 32158 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 03:19:42.378242 32173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:42.379598 32158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:19:42.379784 32158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "cef11dfd8e50422fbe937df7e4261a6c"
format_stamp: "Formatted at 2026-09-24 03:18:56 on dist-test-slave-vd0p"
server_key: "ed99e06f3dde8e89df622d7b84c2dd68"
server_key_iv: "f7175b97f319b79841312996f9abf6ae"
server_key_version: "encryptionkey@0"
I20260924 03:19:42.380738 32158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:42.405610 32158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:42.410084 32158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:42.410403 32158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:42.411485 32158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:19:42.412897 32158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:19:42.413048 32158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:42.413188 32158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:19:42.413298 32158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:42.449664 32158 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:38421
I20260924 03:19:42.449759 32285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:38421 every 8 connection(s)
I20260924 03:19:42.451165 32158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:19:42.451614 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 32158
I20260924 03:19:42.456161 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:42.454342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:36471)
0924 03:19:42.455073 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:42.455330 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":316,"threads_started":1}
W20260924 03:19:42.456794 32286 heartbeater.cc:644] Failed to heartbeat to 127.14.227.190:38957 (0 consecutive failures): Network error: Failed to ping master at 127.14.227.190:38957: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
I20260924 03:19:42.457516 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:42.457232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:44743)
0924 03:19:42.457355 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:42.457418 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 03:19:42.458413 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:42.458116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:51989)
0924 03:19:42.458234 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:42.458304 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 03:19:42.458711 32286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:19:42.574892 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:42.573882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:44017)
0924 03:19:42.574498 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:42.574640 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":325,"threads_started":1}
I20260924 03:19:43.009021 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:43.008231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:38357)
0924 03:19:43.008680 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:43.008793 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":221,"threads_started":1}
I20260924 03:19:43.375720 32292 raft_consensus.cc:482] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:19:43.376008 32292 raft_consensus.cc:504] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:43.377473 32292 leader_election.cc:288] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
Sep 24 03:19:43 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219981, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
Sep 24 03:19:43 dist-test-slave-vd0p krb5kdc[30705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219981, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:19:43.404675 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:43.378096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:55435 (local address 127.14.227.130:35029)
0924 03:19:43.378741 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:43.378759 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:43.378811 (+ 52us) server_negotiation.cc:407] Connection header received
0924 03:19:43.381797 (+ 2986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:43.381810 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:43.382367 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:43.382747 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:43.384533 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:43.385805 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:43.388795 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:43.389438 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:43.389647 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:43.397374 (+ 7727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:43.397388 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:43.397409 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:43.397456 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:43.400393 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:43.401339 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:43.401345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:43.401351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:43.401543 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:43.402013 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:43.402018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:43.402021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:43.402690 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:43.402880 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:43.403890 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0924 03:19:43.404280 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":321,"threads_started":1}
I20260924 03:19:43.404675 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:43.377820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:35029 (local address 127.14.227.129:55435)
0924 03:19:43.378445 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:43.378485 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:43.379097 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:43.383349 (+ 4252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:43.383353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:43.383952 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:19:43.384343 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:43.384363 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:43.386019 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:43.386022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:43.388617 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:43.388630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:43.389625 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:43.389632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:43.389902 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:43.394356 (+ 4454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:19:43.394386 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:19:43.397006 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:19:43.400536 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:43.400558 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:43.400574 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:43.401179 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:43.401663 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:43.401666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:43.401669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:43.401848 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:43.402971 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:19:43.402977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:19:43.403352 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 03:19:43.403642 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 03:19:43.404149 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":207,"spinlock_wait_cycles":5120,"thread_start_us":337,"threads_started":1}
I20260924 03:19:43.404960 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:43.379042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:54685 (local address 127.14.227.131:38421)
0924 03:19:43.379705 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:43.379721 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:43.379770 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:19:43.382065 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:43.382075 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:43.382435 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:43.382757 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:43.384539 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:43.385816 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:43.389878 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:43.390725 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:43.390923 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:43.397686 (+ 6763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:43.397694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:43.397706 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:43.397746 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:43.401416 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:43.402208 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:43.402216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:43.402223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:43.402436 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:43.402825 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:43.402829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:43.402832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:43.403433 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:43.403619 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:43.404174 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 03:19:43.404572 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":338,"threads_started":1}
I20260924 03:19:43.404767 32295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:43.378618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:38421 (local address 127.14.227.129:54685)
0924 03:19:43.379368 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:43.379389 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:19:43.379514 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:19:43.382835 (+ 3321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:19:43.382843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:19:43.383531 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:19:43.384345 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:43.384364 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:43.386014 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:43.386017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:43.388614 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:19:43.388631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:19:43.390914 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:19:43.390921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:19:43.391083 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:43.394347 (+ 3264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:19:43.394370 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:19:43.397431 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:19:43.401543 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:43.401549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:43.401552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:43.402050 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:43.402581 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:19:43.402585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:19:43.402587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:19:43.402706 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:19:43.403568 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:19:43.403572 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:19:43.403729 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 03:19:43.403897 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 03:19:43.404306 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":314,"spinlock_wait_cycles":8576,"thread_start_us":353,"threads_started":1}
W20260924 03:19:43.408378 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
W20260924 03:19:43.408874 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
I20260924 03:19:43.408991 31909 leader_election.cc:302] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271; no voters: 8c603e77e6a14db6883a712ee08493af, cef11dfd8e50422fbe937df7e4261a6c
I20260924 03:19:43.409317 32292 raft_consensus.cc:2726] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:19:43.460117 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:43.459287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:52891)
0924 03:19:43.459747 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:43.459898 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":230,"threads_started":1}
I20260924 03:19:43.576304 32295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:43.575808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:38757)
0924 03:19:43.576069 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:43.576170 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 03:19:44.010550 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:44.009842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:35931)
0924 03:19:44.010314 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:44.010397 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":278,"threads_started":1}
I20260924 03:19:44.461603 32299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:44.460983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:36243)
0924 03:19:44.461374 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:44.461451 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":209,"threads_started":1}
I20260924 03:19:44.578073 32300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:44.577306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:43351)
0924 03:19:44.577767 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:44.577847 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260924 03:19:44.984601 32301 raft_consensus.cc:482] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:19:44.984788 32301 raft_consensus.cc:504] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:44.985471 32301 leader_election.cc:288] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
W20260924 03:19:44.986598 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
W20260924 03:19:44.986855 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
I20260924 03:19:44.986972 31909 leader_election.cc:302] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271; no voters: 8c603e77e6a14db6883a712ee08493af, cef11dfd8e50422fbe937df7e4261a6c
I20260924 03:19:44.987253 32301 raft_consensus.cc:2726] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:19:45.011938 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:45.011316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:56401)
0924 03:19:45.011703 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:45.011781 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":209,"threads_started":1}
I20260924 03:19:45.463038 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:45.462395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:45929)
0924 03:19:45.462811 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:45.462886 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":225,"threads_started":1}
I20260924 03:19:45.579649 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:45.578917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:34927)
0924 03:19:45.579380 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:45.579457 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":265,"threads_started":1}
I20260924 03:19:46.013480 32305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:46.012799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:37239)
0924 03:19:46.013216 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:46.013298 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":213,"threads_started":1}
I20260924 03:19:46.464659 32306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:46.463877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:43507)
0924 03:19:46.464408 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:46.464486 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":285,"threads_started":1}
I20260924 03:19:46.581408 32307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:46.580676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:34081)
0924 03:19:46.581135 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:46.581211 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260924 03:19:47.015126 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:47.014425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:48951)
0924 03:19:47.014870 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:47.014954 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":240,"threads_started":1}
I20260924 03:19:47.073727 32309 raft_consensus.cc:482] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:19:47.074023 32309 raft_consensus.cc:504] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:47.074812 32309 leader_election.cc:288] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
W20260924 03:19:47.075650 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
W20260924 03:19:47.075868 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
I20260924 03:19:47.076009 31909 leader_election.cc:302] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271; no voters: 8c603e77e6a14db6883a712ee08493af, cef11dfd8e50422fbe937df7e4261a6c
I20260924 03:19:47.076261 32309 raft_consensus.cc:2726] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:19:47.466368 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:47.465627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:36729)
0924 03:19:47.466146 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:47.466219 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":297,"threads_started":1}
I20260924 03:19:47.583168 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:47.582345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:43035)
0924 03:19:47.582856 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:47.582968 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":274,"threads_started":1}
I20260924 03:19:48.016801 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:48.016036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:43173)
0924 03:19:48.016523 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:48.016611 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
I20260924 03:19:48.468021 32313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:48.467170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:37887)
0924 03:19:48.467727 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:48.467843 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":277,"threads_started":1}
I20260924 03:19:48.584889 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:48.584098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:51963)
0924 03:19:48.584566 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:48.584692 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260924 03:19:49.018525 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:49.017695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:50011)
0924 03:19:49.018236 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:49.018325 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":271,"threads_started":1}
I20260924 03:19:49.470166 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:49.469053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:53587)
0924 03:19:49.469747 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:49.469856 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":330,"threads_started":1}
I20260924 03:19:49.586462 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:49.585672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:39121)
0924 03:19:49.586200 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:49.586280 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":295,"threads_started":1}
I20260924 03:19:50.020149 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:50.019396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:49625)
0924 03:19:50.019872 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:50.019957 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":266,"threads_started":1}
I20260924 03:19:50.031345 32319 raft_consensus.cc:482] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:19:50.031600 32319 raft_consensus.cc:504] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:50.032266 32319 leader_election.cc:288] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
W20260924 03:19:50.033185 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
W20260924 03:19:50.033358 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
I20260924 03:19:50.033429 31909 leader_election.cc:302] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271; no voters: 8c603e77e6a14db6883a712ee08493af, cef11dfd8e50422fbe937df7e4261a6c
I20260924 03:19:50.033666 32319 raft_consensus.cc:2726] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:19:50.471797 32320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:50.470997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:56725)
0924 03:19:50.471474 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:50.471578 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":280,"threads_started":1}
I20260924 03:19:50.588022 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:50.587288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:52259)
0924 03:19:50.587760 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:50.587837 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":237,"threads_started":1}
I20260924 03:19:51.021896 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:51.021103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:53159)
0924 03:19:51.021614 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:51.021709 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":241,"threads_started":1}
I20260924 03:19:51.473742 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:51.472743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:40505)
0924 03:19:51.473355 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:51.473488 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":296,"threads_started":1}
I20260924 03:19:51.590009 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:51.589187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:60933)
0924 03:19:51.589717 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:51.589794 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":309,"threads_started":1}
I20260924 03:19:52.023576 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:52.022863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:46305)
0924 03:19:52.023315 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:52.023397 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":255,"threads_started":1}
I20260924 03:19:52.475396 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:52.474704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:46281)
0924 03:19:52.475120 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:52.475201 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
I20260924 03:19:52.591662 32327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:52.590936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:38681)
0924 03:19:52.591393 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:52.591480 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":257,"threads_started":1}
I20260924 03:19:53.025259 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:53.024509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:51007)
0924 03:19:53.025013 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:53.025097 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
I20260924 03:19:53.477085 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:53.476243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:32841)
0924 03:19:53.476758 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:53.476871 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260924 03:19:53.593308 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:53.592610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:60579)
0924 03:19:53.593065 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:53.593145 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":253,"threads_started":1}
I20260924 03:19:54.027132 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:54.026278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:59049)
0924 03:19:54.026764 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:54.026866 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":231,"threads_started":1}
I20260924 03:19:54.479061 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:54.478150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:44675)
0924 03:19:54.478691 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:54.478816 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
I20260924 03:19:54.595013 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:54.594289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:52795)
0924 03:19:54.594733 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:54.594841 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":233,"threads_started":1}
I20260924 03:19:55.028712 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:55.028028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:40607)
0924 03:19:55.028473 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:55.028558 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":207,"threads_started":1}
I20260924 03:19:55.480798 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:55.480076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:53695)
0924 03:19:55.480545 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:55.480638 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":245,"threads_started":1}
I20260924 03:19:55.596679 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:55.595895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:36989)
0924 03:19:55.596388 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:55.596481 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
I20260924 03:19:56.030369 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:56.029626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:39491)
0924 03:19:56.030082 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:56.030159 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":214,"threads_started":1}
I20260924 03:19:56.482383 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:56.481604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:41589)
0924 03:19:56.482084 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:56.482175 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":225,"threads_started":1}
I20260924 03:19:56.598588 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:56.597814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.129:57963)
0924 03:19:56.598313 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:56.598398 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":282,"threads_started":1}
I20260924 03:19:56.960687 32340 raft_consensus.cc:482] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:19:56.960924 32340 raft_consensus.cc:504] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c603e77e6a14db6883a712ee08493af" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35029 } } peers { permanent_uuid: "87da36cd174d4ec1af9b20b7d0f51271" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45663 } } peers { permanent_uuid: "cef11dfd8e50422fbe937df7e4261a6c" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 38421 } }
I20260924 03:19:56.961632 32340 leader_election.cc:288] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029), cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421)
W20260924 03:19:56.962735 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c603e77e6a14db6883a712ee08493af (127.14.227.130:35029): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
W20260924 03:19:56.962944 31909 leader_election.cc:341] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cef11dfd8e50422fbe937df7e4261a6c (127.14.227.131:38421): Not found: Tablet not found: 702b4a0509ac4c4b94037ce3bfe5b9bd
I20260924 03:19:56.963104 31909 leader_election.cc:302] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87da36cd174d4ec1af9b20b7d0f51271; no voters: 8c603e77e6a14db6883a712ee08493af, cef11dfd8e50422fbe937df7e4261a6c
I20260924 03:19:56.963367 32340 raft_consensus.cc:2726] T 702b4a0509ac4c4b94037ce3bfe5b9bd P 87da36cd174d4ec1af9b20b7d0f51271 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:19:57.032168 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:57.031410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.130:33095)
0924 03:19:57.031864 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:57.031979 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":210,"threads_started":1}
I20260924 03:19:57.457202 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 31853
I20260924 03:19:57.477159 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:57.458511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.14.227.129:45663)
0924 03:19:57.459193 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:57.459208 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:57.459435 (+ 227us) server_negotiation.cc:407] Connection header received
0924 03:19:57.459638 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:57.459646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:57.459873 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:57.460007 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:57.462083 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:57.463064 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:57.464500 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:57.465065 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:57.465162 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:57.471759 (+ 6597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:57.471772 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:57.471786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:57.471822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:57.474773 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:57.475420 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:57.475428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:57.475430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:57.475555 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:57.475874 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:57.475878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:57.475879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:57.476227 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:57.476464 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:57.476690 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 03:19:57.476889 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":270,"threads_started":1}
I20260924 03:19:57.478431 32018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:19:57.485631 32344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:57.483490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:38957 (local address 127.14.227.131:57235)
0924 03:19:57.485370 (+ 1880us) negotiation.cc:107] Waiting for socket to connect
0924 03:19:57.485480 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.190:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1762,"thread_start_us":1641,"threads_started":1}
I20260924 03:19:57.567656 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 31853
I20260924 03:19:57.584659 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 32027
I20260924 03:19:57.598770 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:57.585476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.14.227.130:35029)
0924 03:19:57.586043 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:57.586051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:57.586080 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:19:57.586150 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:57.586154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:57.586219 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:57.586343 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:57.587926 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:57.588697 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:57.590015 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:57.590480 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:57.590625 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:57.593347 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:57.593353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:57.593362 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:57.593396 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:57.596007 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:57.596745 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:57.596751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:57.596755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:57.596897 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:57.597284 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:57.597290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:57.597293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:57.597545 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:57.597852 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:57.598176 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 03:19:57.598413 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
I20260924 03:19:57.599435 32150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:19:57.680655 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 32027
I20260924 03:19:57.699409 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 32158
I20260924 03:19:57.712160 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:57.700240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.14.227.131:38421)
0924 03:19:57.700875 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:57.700879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:57.700897 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:19:57.700952 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:57.700955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:57.701025 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:57.701118 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:57.702481 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:57.703269 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:57.704472 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:57.704877 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:57.704978 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:57.707566 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:57.707571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:57.707574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:57.707606 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:57.710036 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:57.710587 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:57.710590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:57.710592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:57.710771 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:57.711053 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:57.711055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:57.711057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:57.711315 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:57.711519 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:57.711680 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 03:19:57.711848 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":310,"threads_started":1}
I20260924 03:19:57.712828 32281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:19:57.795887 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 32158
2026-09-24T03:19:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64202 ms)
[----------] 30 tests from SecurityITest (290433 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:19:57 dist-test-slave-vd0p krb5kdc[32351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:19:57 dist-test-slave-vd0p krb5kdc[32351](info): set up 2 sockets
Sep 24 03:19:57 dist-test-slave-vd0p krb5kdc[32351](info): commencing operation
krb5kdc: starting...
W20260924 03:19:58.886512 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:19:59 dist-test-slave-vd0p krb5kdc[32351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:19:59.127345 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37879
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:19:59.459995 32364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:19:59.460261 32364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:19:59.460472 32364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:19:59.467341 32364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:19:59.467446 32364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:19:59.467550 32364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:19:59.467664 32364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:19:59.467727 32364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:19:59.476938 32364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:19:59.478229 32364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:19:59.479571 32364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:19:59.491227 32369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:59.491326 32372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:19:59.491351 32370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:19:59.492251 32364 server_base.cc:1034] running on GCE node
I20260924 03:19:59.493068 32364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:19:59.494620 32364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:19:59.495867 32364 hybrid_clock.cc:648] HybridClock initialized: now 1790219999495809 us; error 36 us; skew 500 ppm
I20260924 03:19:59.496248 32364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:19:59 dist-test-slave-vd0p krb5kdc[32351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219999, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:19:59.499457 32364 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:19:59.500890 32364 webserver.cc:460] Webserver started at http://127.14.227.190:42677/ using document root <none> and password file <none>
I20260924 03:19:59.501613 32364 fs_manager.cc:359] Metadata directory not provided
I20260924 03:19:59.501770 32364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:19:59.502141 32364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:19:59.505012 32364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c"
format_stamp: "Formatted at 2026-09-24 03:19:59 on dist-test-slave-vd0p"
server_key: "32160c2af541e243ff9bac3f5abfaf4d"
server_key_iv: "6afcec91d2bb237d0a10a88c10980cde"
server_key_version: "encryptionkey@0"
I20260924 03:19:59.505743 32364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c"
format_stamp: "Formatted at 2026-09-24 03:19:59 on dist-test-slave-vd0p"
server_key: "32160c2af541e243ff9bac3f5abfaf4d"
server_key_iv: "6afcec91d2bb237d0a10a88c10980cde"
server_key_version: "encryptionkey@0"
I20260924 03:19:59.510654 32364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 03:19:59.514259 32379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:19:59.515486 32364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:19:59.515681 32364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c"
format_stamp: "Formatted at 2026-09-24 03:19:59 on dist-test-slave-vd0p"
server_key: "32160c2af541e243ff9bac3f5abfaf4d"
server_key_iv: "6afcec91d2bb237d0a10a88c10980cde"
server_key_version: "encryptionkey@0"
I20260924 03:19:59.515868 32364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:19:59.532567 32364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:19:59.536502 32364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:19:59.536790 32364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:19:59.554821 32364 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:37879
I20260924 03:19:59.554780 32430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:37879 every 8 connection(s)
I20260924 03:19:59.556133 32364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:19:59.558151 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 32364
I20260924 03:19:59.558367 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:19:59.559039 15246 external_mini_cluster.cc:1428] Setting key 183c2600df6bc869d5b1861570958567
I20260924 03:19:59.559907 32431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:19:59 dist-test-slave-vd0p krb5kdc[32351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:19:59.573642 32431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c: Bootstrap starting.
I20260924 03:19:59.577615 32431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c: Neither blocks nor log segments found. Creating new log.
I20260924 03:19:59.578995 32431 log.cc:824] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:19:59.582350 32431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c: No bootstrap required, opened a new log
I20260924 03:19:59.583491 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:19:59.560963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.14.227.190:37879)
0924 03:19:59.561762 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 03:19:59.561778 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:19:59.561825 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:19:59.563577 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:19:59.563600 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:19:59.564056 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:19:59.564459 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:19:59.565804 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:59.567813 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:59.569253 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:19:59.570325 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:19:59.570526 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:19:59.574818 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:19:59.574827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:19:59.574845 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:19:59.574888 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:19:59.578892 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:59.579737 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:59.579746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:59.579751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:59.579915 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:19:59.580387 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:19:59.580392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:19:59.580395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:19:59.581092 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:19:59.581368 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 03:19:59.581743 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 03:19:59.582188 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":349,"threads_started":1}
I20260924 03:19:59.590754 32431 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37879 } }
I20260924 03:19:59.591071 32431 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:19:59.591147 32431 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecea0b64e8fa49aa97e6b2f16fcb331c, State: Initialized, Role: FOLLOWER
I20260924 03:19:59.591652 32431 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37879 } }
I20260924 03:19:59.591847 32431 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:19:59.591948 32431 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:19:59.592072 32431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:19:59.594430 32431 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37879 } }
I20260924 03:19:59.594849 32431 leader_election.cc:302] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecea0b64e8fa49aa97e6b2f16fcb331c; no voters:
I20260924 03:19:59.595330 32431 leader_election.cc:288] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:19:59.595480 32436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:19:59.596160 32436 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [term 1 LEADER]: Becoming Leader. State: Replica: ecea0b64e8fa49aa97e6b2f16fcb331c, State: Running, Role: LEADER
I20260924 03:19:59.596702 32436 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37879 } }
I20260924 03:19:59.597324 32431 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:19:59.599808 32437 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37879 } } }
I20260924 03:19:59.600188 32437 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [sys.catalog]: This master's current role is: LEADER
I20260924 03:19:59.599596 32438 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecea0b64e8fa49aa97e6b2f16fcb331c. Latest consensus state: current_term: 1 leader_uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecea0b64e8fa49aa97e6b2f16fcb331c" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 37879 } } }
I20260924 03:19:59.600615 32438 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c [sys.catalog]: This master's current role is: LEADER
W20260924 03:19:59.607890 32452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:19:59.607990 32452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:19:59.608116 32445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:19:59.610229 32445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:19:59.619032 32445 catalog_manager.cc:1347] Generated new cluster ID: 540380b559b34096a2a552701b2a0c95
I20260924 03:19:59.619153 32445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:19:59.631374 32445 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:19:59.632594 32445 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:19:59.640964 32445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c: Generated new TSK 0
I20260924 03:19:59.641744 32445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:19:59.693117 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37879
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:00.010888 32457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:00.011161 32457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:00.011333 32457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:00.018610 32457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:00.018711 32457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:00.018893 32457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:20:00.028584 32457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:00.029764 32457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:00.031013 32457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:00.043771 32465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:00.043929 32462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:00.043815 32463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:00.044616 32457 server_base.cc:1034] running on GCE node
I20260924 03:20:00.045166 32457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:00.046156 32457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:00.047367 32457 hybrid_clock.cc:648] HybridClock initialized: now 1790220000047329 us; error 51 us; skew 500 ppm
I20260924 03:20:00.047665 32457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:00 dist-test-slave-vd0p krb5kdc[32351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220000, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:00.050696 32457 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:20:00.052093 32457 webserver.cc:460] Webserver started at http://127.14.227.129:44541/ using document root <none> and password file <none>
I20260924 03:20:00.052734 32457 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:00.052911 32457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:00.053202 32457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:00.056131 32457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "5467410629634e188b34c3b70fd2f631"
format_stamp: "Formatted at 2026-09-24 03:20:00 on dist-test-slave-vd0p"
server_key: "7461729cb2b671b7f8636313a310d847"
server_key_iv: "2412a79d28c1b53d0173ed87c0d9acad"
server_key_version: "encryptionkey@0"
I20260924 03:20:00.056888 32457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "5467410629634e188b34c3b70fd2f631"
format_stamp: "Formatted at 2026-09-24 03:20:00 on dist-test-slave-vd0p"
server_key: "7461729cb2b671b7f8636313a310d847"
server_key_iv: "2412a79d28c1b53d0173ed87c0d9acad"
server_key_version: "encryptionkey@0"
I20260924 03:20:00.061738 32457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260924 03:20:00.064914 32472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:00.066169 32457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:00.066319 32457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "5467410629634e188b34c3b70fd2f631"
format_stamp: "Formatted at 2026-09-24 03:20:00 on dist-test-slave-vd0p"
server_key: "7461729cb2b671b7f8636313a310d847"
server_key_iv: "2412a79d28c1b53d0173ed87c0d9acad"
server_key_version: "encryptionkey@0"
I20260924 03:20:00.066486 32457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:00.098518 32457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:00.102823 32457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:00.103127 32457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:00.104117 32457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:00.105460 32457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:00.105571 32457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:00.105711 32457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:00.105778 32457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:00.138039 32457 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:33079
I20260924 03:20:00.138118 32584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:33079 every 8 connection(s)
I20260924 03:20:00.139369 32457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:20:00.145605 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 32457
I20260924 03:20:00.145857 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:20:00.146539 15246 external_mini_cluster.cc:1428] Setting key 5e4b58b6989c5b9dd2494939893af26d
Sep 24 03:20:00 dist-test-slave-vd0p krb5kdc[32351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220000, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:00.160295 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:00.142083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:44423 (local address 127.14.227.190:37879)
0924 03:20:00.142515 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:00.142520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:00.143546 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 03:20:00.145310 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:00.145317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:00.145405 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:00.145538 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:00.147160 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:00.148088 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:00.149596 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:00.150055 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:00.150187 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:00.153686 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:00.153691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:00.153694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:00.153725 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:00.157094 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:00.157841 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:00.157850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:00.157856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:00.158025 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:00.158413 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:00.158420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:00.158426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:00.158804 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:00.158939 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:00.159585 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 03:20:00.159830 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260924 03:20:00.160926 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:00.142682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37879 (local address 127.14.227.129:44423)
0924 03:20:00.143375 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:00.143413 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:00.145044 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:00.145772 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:00.145780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:00.146373 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:00.146980 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:00.147007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:00.148247 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:00.148252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:00.149447 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:00.149458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:00.150284 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:00.150291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:00.150506 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:00.151163 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:00.151187 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:00.153470 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:00.157237 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:00.157243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:00.157256 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:00.157701 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:00.158135 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:00.158138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:00.158140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:00.158311 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:00.158925 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:00.158931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:00.159226 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 03:20:00.159450 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 03:20:00.159824 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":311,"threads_started":1}
I20260924 03:20:00.162474 32585 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37879
I20260924 03:20:00.162806 32585 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:00.163470 32585 heartbeater.cc:511] Master 127.14.227.190:37879 requested a full tablet report, sending...
I20260924 03:20:00.165473 32396 ts_manager.cc:194] Registered new tserver with Master: 5467410629634e188b34c3b70fd2f631 (127.14.227.129:33079)
I20260924 03:20:00.167047 32396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:44423
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:20:00.199612 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37879
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:00.527472 32591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:00.527813 32591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:00.527999 32591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:00.535187 32591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:00.535332 32591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:00.535539 32591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:20:00.545032 32591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:00.546375 32591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:00.547806 32591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:00.561494 32596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:00.561659 32599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:00.561616 32597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:00.563428 32591 server_base.cc:1034] running on GCE node
I20260924 03:20:00.564134 32591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:00.565167 32591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:00.566371 32591 hybrid_clock.cc:648] HybridClock initialized: now 1790220000566314 us; error 48 us; skew 500 ppm
I20260924 03:20:00.566735 32591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:00 dist-test-slave-vd0p krb5kdc[32351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220000, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:00.570066 32591 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:20:00.572878 32591 webserver.cc:460] Webserver started at http://127.14.227.130:40237/ using document root <none> and password file <none>
I20260924 03:20:00.573559 32591 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:00.573696 32591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:00.574066 32591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:00.576886 32591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "8648e5461d8a4162b7c8bd1692bab8be"
format_stamp: "Formatted at 2026-09-24 03:20:00 on dist-test-slave-vd0p"
server_key: "ff0e3ef84fa405331546ba9bfaaeb8c9"
server_key_iv: "f88b10e5b9214511f8ba2d0c3b2220bf"
server_key_version: "encryptionkey@0"
I20260924 03:20:00.577638 32591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "8648e5461d8a4162b7c8bd1692bab8be"
format_stamp: "Formatted at 2026-09-24 03:20:00 on dist-test-slave-vd0p"
server_key: "ff0e3ef84fa405331546ba9bfaaeb8c9"
server_key_iv: "f88b10e5b9214511f8ba2d0c3b2220bf"
server_key_version: "encryptionkey@0"
I20260924 03:20:00.582827 32591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:20:00.586062 32606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:00.587252 32591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:20:00.587414 32591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "8648e5461d8a4162b7c8bd1692bab8be"
format_stamp: "Formatted at 2026-09-24 03:20:00 on dist-test-slave-vd0p"
server_key: "ff0e3ef84fa405331546ba9bfaaeb8c9"
server_key_iv: "f88b10e5b9214511f8ba2d0c3b2220bf"
server_key_version: "encryptionkey@0"
I20260924 03:20:00.587600 32591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:00.599349 32591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:00.603592 32591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:00.603906 32591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:00.604861 32591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:00.606313 32591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:00.606386 32591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:00.606539 32591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:00.606590 32591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:00.639904 32591 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:37083
I20260924 03:20:00.639954 32718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:37083 every 8 connection(s)
I20260924 03:20:00.641355 32591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:20:00.642072 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 32591
I20260924 03:20:00.642285 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:20:00.642874 15246 external_mini_cluster.cc:1428] Setting key d52414d2658e2f193f6c90b1d08492e3
Sep 24 03:20:00 dist-test-slave-vd0p krb5kdc[32351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220000, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:00.663008 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:00.644137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:35333 (local address 127.14.227.190:37879)
0924 03:20:00.644341 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:00.644347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:00.645734 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 03:20:00.647561 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:00.647568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:00.647655 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:00.647765 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:00.649186 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:00.650342 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:00.651854 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:00.652461 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:00.652556 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:00.657179 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:00.657184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:00.657187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:00.657219 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:00.659740 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:00.660652 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:00.660656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:00.660658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:00.660780 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:00.661231 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:00.661234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:00.661236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:00.661592 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:00.661733 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:00.662594 (+ 861us) server_negotiation.cc:299] Negotiation successful
0924 03:20:00.662770 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:20:00.664670 32721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:00.644753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37879 (local address 127.14.227.130:35333)
0924 03:20:00.645569 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:00.645611 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:00.647300 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:00.647973 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:00.647980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:00.648487 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:00.649011 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:00.649028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:00.650484 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:00.650487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:00.651712 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:00.651723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:00.652692 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:00.652700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:00.653010 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:00.653802 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:00.653840 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:00.656949 (+ 3109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:00.659873 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:00.659884 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:00.659903 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:00.660536 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:00.660880 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:00.660886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:00.660893 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:00.661127 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:00.661733 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:00.661744 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:00.662211 (+ 467us) client_negotiation.cc:769] Sending connection context
0924 03:20:00.662542 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 03:20:00.663119 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":321,"threads_started":1}
I20260924 03:20:00.666177 32719 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37879
I20260924 03:20:00.666467 32719 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:00.667174 32719 heartbeater.cc:511] Master 127.14.227.190:37879 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
I20260924 03:20:00.668687 32396 ts_manager.cc:194] Registered new tserver with Master: 8648e5461d8a4162b7c8bd1692bab8be (127.14.227.130:37083)
I20260924 03:20:00.669539 32396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:35333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:20:00.697818 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37879
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:01.016470 32724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:01.016781 32724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:01.016953 32724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:01.024238 32724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:01.024374 32724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:01.024552 32724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:20:01.033974 32724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:37879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:01.035156 32724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:01.036382 32724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:01.049271 32732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:01.049414 32729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:01.049356 32730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:01.050462 32724 server_base.cc:1034] running on GCE node
I20260924 03:20:01.051046 32724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:01.051956 32724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:01.053182 32724 hybrid_clock.cc:648] HybridClock initialized: now 1790220001053146 us; error 47 us; skew 500 ppm
I20260924 03:20:01.053498 32724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:01 dist-test-slave-vd0p krb5kdc[32351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220001, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:01.056730 32724 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:20:01.058290 32724 webserver.cc:460] Webserver started at http://127.14.227.131:40295/ using document root <none> and password file <none>
I20260924 03:20:01.058966 32724 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:01.059108 32724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:01.059397 32724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:01.062242 32724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "58c97a6afa064f2e8c45cd82d942fdf3"
format_stamp: "Formatted at 2026-09-24 03:20:01 on dist-test-slave-vd0p"
server_key: "15e3503257714606778cf8d04af43993"
server_key_iv: "3354d4f2a678dc7db77105447a6efdf5"
server_key_version: "encryptionkey@0"
I20260924 03:20:01.063278 32724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "58c97a6afa064f2e8c45cd82d942fdf3"
format_stamp: "Formatted at 2026-09-24 03:20:01 on dist-test-slave-vd0p"
server_key: "15e3503257714606778cf8d04af43993"
server_key_iv: "3354d4f2a678dc7db77105447a6efdf5"
server_key_version: "encryptionkey@0"
I20260924 03:20:01.067858 32724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 03:20:01.071033 32739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:01.072307 32724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 03:20:01.072470 32724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "58c97a6afa064f2e8c45cd82d942fdf3"
format_stamp: "Formatted at 2026-09-24 03:20:01 on dist-test-slave-vd0p"
server_key: "15e3503257714606778cf8d04af43993"
server_key_iv: "3354d4f2a678dc7db77105447a6efdf5"
server_key_version: "encryptionkey@0"
I20260924 03:20:01.072662 32724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:01.105540 32724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:01.109282 32724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:01.109552 32724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:01.110486 32724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:01.111714 32724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:01.111816 32724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:01.111966 32724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:01.112037 32724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:01.144608 32724 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:36767
I20260924 03:20:01.144630 383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:36767 every 8 connection(s)
I20260924 03:20:01.145926 32724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:20:01.149749 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 32724
I20260924 03:20:01.149931 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:20:01.150377 15246 external_mini_cluster.cc:1428] Setting key 3fc97a187d5b6c2c5da6d2fa60de13b9
Sep 24 03:20:01 dist-test-slave-vd0p krb5kdc[32351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220001, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:01.165000 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.148255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:43489 (local address 127.14.227.190:37879)
0924 03:20:01.148473 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.148480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.149578 (+ 1098us) server_negotiation.cc:407] Connection header received
0924 03:20:01.151657 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.151663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.151760 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.151842 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:01.153615 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.154431 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.155977 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.156373 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.156482 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.159657 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:01.159662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:01.159665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:01.159728 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:01.162350 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.162964 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.162968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.162970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.163084 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.163411 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.163414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.163415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.163642 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:01.163754 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.164495 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.164642 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:20:01.166066 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.148787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:37879 (local address 127.14.227.131:43489)
0924 03:20:01.149422 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:01.149473 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:01.151311 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:01.152113 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:01.152125 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:01.152783 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:01.153443 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.153469 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.154665 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.154669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.155858 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.155869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.156620 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.156631 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.156854 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.157407 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:01.157432 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:01.159489 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:01.162456 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:01.162461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:01.162471 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:01.162873 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:01.163171 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:01.163174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:01.163176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:01.163329 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:01.163762 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:01.163771 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:01.164111 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 03:20:01.164395 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 03:20:01.164885 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":257,"threads_started":1}
I20260924 03:20:01.167337 384 heartbeater.cc:347] Connected to a master server at 127.14.227.190:37879
I20260924 03:20:01.167572 384 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:01.168162 384 heartbeater.cc:511] Master 127.14.227.190:37879 requested a full tablet report, sending...
I20260924 03:20:01.169387 32396 ts_manager.cc:194] Registered new tserver with Master: 58c97a6afa064f2e8c45cd82d942fdf3 (127.14.227.131:36767)
I20260924 03:20:01.169883 32585 heartbeater.cc:503] Master 127.14.227.190:37879 was elected leader, sending a full tablet report...
I20260924 03:20:01.170327 32396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:43489
I20260924 03:20:01.177716 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:20:01.194356 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.182198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.14.227.190:37879)
0924 03:20:01.182381 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.182389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.182836 (+ 447us) server_negotiation.cc:407] Connection header received
0924 03:20:01.182955 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.182958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.183013 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.183100 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:01.184437 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.185131 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.186358 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.186736 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.186837 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.189039 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:01.189043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:01.189047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:01.189074 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:01.191720 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.192362 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.192367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.192369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.192544 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.192987 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.192991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.192993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.193289 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:01.193466 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.193814 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.194034 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:20:01.198375 32395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:20:01.205433 32395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:20:01.229544 398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.217417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.14.227.131:36767)
0924 03:20:01.219826 (+ 2409us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.219834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.219866 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:20:01.220091 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.220104 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.220475 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.221644 (+ 1169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:01.222011 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.224036 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.227039 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.228454 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.228671 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.228843 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.229077 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.229216 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2224,"thread_start_us":2082,"threads_started":1}
I20260924 03:20:01.229578 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.217097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:36767 (local address 127.0.0.1:39728)
0924 03:20:01.218148 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:01.218202 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:01.218409 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:01.220753 (+ 2344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:01.220764 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:01.220817 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:01.221370 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.221385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.224227 (+ 2842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.224235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.226874 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.226885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.228668 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.228674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.228848 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.228908 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 03:20:01.229039 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 03:20:01.229216 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":7,"thread_start_us":2309,"threads_started":1}
I20260924 03:20:01.230602 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.218045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.14.227.129:33079)
0924 03:20:01.219181 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.219193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.219860 (+ 667us) server_negotiation.cc:407] Connection header received
0924 03:20:01.220193 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.220201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.220493 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.220702 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:01.222004 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.224064 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.227771 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.229270 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.229439 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.229580 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.230122 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.230292 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":999,"threads_started":1}
I20260924 03:20:01.231331 396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.217337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:33079 (local address 127.0.0.1:50102)
0924 03:20:01.219731 (+ 2394us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:01.219766 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:01.219991 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:01.221101 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:01.221109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:01.221149 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:01.221763 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.221779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.224274 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.224282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.227602 (+ 3320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.227616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.229737 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.229744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.229901 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.229927 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 03:20:01.230857 (+ 930us) client_negotiation.cc:240] Negotiation successful
0924 03:20:01.231002 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2220,"mutex_wait_us":421,"thread_start_us":934,"threads_started":1}
I20260924 03:20:01.233692 319 tablet_service.cc:1467] Processing CreateTablet for tablet 5d7eb519359940e486441afbe8828b72 (DEFAULT_TABLE table=test-table [id=d5b5cdd0e12c4566a33fbe83b3bf5c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:01.234500 32520 tablet_service.cc:1467] Processing CreateTablet for tablet 5d7eb519359940e486441afbe8828b72 (DEFAULT_TABLE table=test-table [id=d5b5cdd0e12c4566a33fbe83b3bf5c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:01.235248 319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d7eb519359940e486441afbe8828b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:01.235903 32520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d7eb519359940e486441afbe8828b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:01.238089 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.220176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.14.227.130:37083)
0924 03:20:01.225891 (+ 5715us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.225899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.227562 (+ 1663us) server_negotiation.cc:407] Connection header received
0924 03:20:01.227657 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.227665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.227882 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.228091 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:01.228703 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.230605 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.233087 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.235914 (+ 2827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.237334 (+ 1420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.237490 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.237572 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.237743 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5569,"thread_start_us":5455,"threads_started":1}
I20260924 03:20:01.238325 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.219856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:37083 (local address 127.0.0.1:53492)
0924 03:20:01.227327 (+ 7471us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:01.227342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:01.227447 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:01.228093 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:01.228098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:01.228122 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:01.228478 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.228489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.230924 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.230928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.232925 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.232938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.236221 (+ 3283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.236228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.236489 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.236549 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 03:20:01.236692 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 03:20:01.236894 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7343,"thread_start_us":7174,"threads_started":1}
I20260924 03:20:01.241884 32654 tablet_service.cc:1467] Processing CreateTablet for tablet 5d7eb519359940e486441afbe8828b72 (DEFAULT_TABLE table=test-table [id=d5b5cdd0e12c4566a33fbe83b3bf5c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:01.243331 32654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d7eb519359940e486441afbe8828b72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:01.248529 402 tablet_bootstrap.cc:492] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Bootstrap starting.
I20260924 03:20:01.252072 402 tablet_bootstrap.cc:654] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:01.252863 403 tablet_bootstrap.cc:492] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Bootstrap starting.
I20260924 03:20:01.253305 402 log.cc:824] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:01.256198 403 tablet_bootstrap.cc:654] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:01.256781 402 tablet_bootstrap.cc:492] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: No bootstrap required, opened a new log
I20260924 03:20:01.257118 402 ts_tablet_manager.cc:1397] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260924 03:20:01.257314 403 log.cc:824] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:01.258641 405 tablet_bootstrap.cc:492] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Bootstrap starting.
I20260924 03:20:01.260499 403 tablet_bootstrap.cc:492] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: No bootstrap required, opened a new log
I20260924 03:20:01.260742 403 ts_tablet_manager.cc:1397] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 03:20:01.261293 405 tablet_bootstrap.cc:654] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:01.262284 405 log.cc:824] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:01.264837 405 tablet_bootstrap.cc:492] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: No bootstrap required, opened a new log
I20260924 03:20:01.265066 405 ts_tablet_manager.cc:1397] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260924 03:20:01.267751 402 raft_consensus.cc:348] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.268183 402 raft_consensus.cc:374] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:01.268280 402 raft_consensus.cc:729] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c97a6afa064f2e8c45cd82d942fdf3, State: Initialized, Role: FOLLOWER
I20260924 03:20:01.268874 402 consensus_queue.cc:260] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.269970 384 heartbeater.cc:503] Master 127.14.227.190:37879 was elected leader, sending a full tablet report...
I20260924 03:20:01.271082 403 raft_consensus.cc:348] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.271490 403 raft_consensus.cc:374] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:01.271591 403 raft_consensus.cc:729] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5467410629634e188b34c3b70fd2f631, State: Initialized, Role: FOLLOWER
I20260924 03:20:01.271986 402 ts_tablet_manager.cc:1428] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 03:20:01.272255 403 consensus_queue.cc:260] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.273155 405 raft_consensus.cc:348] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.273546 405 raft_consensus.cc:374] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:01.273634 405 raft_consensus.cc:729] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8648e5461d8a4162b7c8bd1692bab8be, State: Initialized, Role: FOLLOWER
I20260924 03:20:01.273802 403 ts_tablet_manager.cc:1428] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260924 03:20:01.274344 405 consensus_queue.cc:260] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.275918 32719 heartbeater.cc:503] Master 127.14.227.190:37879 was elected leader, sending a full tablet report...
I20260924 03:20:01.277709 405 ts_tablet_manager.cc:1428] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
W20260924 03:20:01.391896 32586 tablet.cc:2369] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:20:01.393368 32720 tablet.cc:2369] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:20:01.397506 385 tablet.cc:2369] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:01.498884 409 raft_consensus.cc:482] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:20:01.499171 409 raft_consensus.cc:504] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.500996 409 leader_election.cc:288] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58c97a6afa064f2e8c45cd82d942fdf3 (127.14.227.131:36767), 8648e5461d8a4162b7c8bd1692bab8be (127.14.227.130:37083)
I20260924 03:20:01.508013 398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.501568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:48463 (local address 127.14.227.131:36767)
0924 03:20:01.501764 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.501769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.502400 (+ 631us) server_negotiation.cc:407] Connection header received
0924 03:20:01.502612 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.502618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.502698 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.502811 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:01.503672 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.504462 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.506326 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.507094 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.507292 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.507369 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.507625 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.507746 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:20:01.508180 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.501485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:36767 (local address 127.14.227.129:48463)
0924 03:20:01.502285 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:01.502307 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:01.502452 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:01.503067 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:01.503072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:01.503108 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:01.503529 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.503538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.504611 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.504619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.506179 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.506192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.507307 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.507313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.507470 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.507494 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:20:01.507599 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 03:20:01.507745 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":456,"threads_started":1}
I20260924 03:20:01.508412 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.502685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:47061 (local address 127.14.227.130:37083)
0924 03:20:01.502977 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.502983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.503386 (+ 403us) server_negotiation.cc:407] Connection header received
0924 03:20:01.503595 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.503599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.503653 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.503744 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:01.504277 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.504996 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.506764 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.507516 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.507690 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.507767 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.508052 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.508175 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:20:01.508618 412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.502412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:37083 (local address 127.14.227.129:47061)
0924 03:20:01.503252 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:01.503291 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:01.503449 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:01.503756 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:01.503760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:01.503782 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:01.504116 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.504125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.505133 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.505138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.506610 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:01.506622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.507737 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.507743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:01.507901 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.507923 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:20:01.508023 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 03:20:01.508146 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":525,"threads_started":1}
I20260924 03:20:01.509495 339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d7eb519359940e486441afbe8828b72" candidate_uuid: "5467410629634e188b34c3b70fd2f631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" is_pre_election: true
I20260924 03:20:01.509809 32674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d7eb519359940e486441afbe8828b72" candidate_uuid: "5467410629634e188b34c3b70fd2f631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8648e5461d8a4162b7c8bd1692bab8be" is_pre_election: true
I20260924 03:20:01.510074 339 raft_consensus.cc:2454] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5467410629634e188b34c3b70fd2f631 in term 0.
I20260924 03:20:01.510381 32674 raft_consensus.cc:2454] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5467410629634e188b34c3b70fd2f631 in term 0.
I20260924 03:20:01.510798 32473 leader_election.cc:302] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5467410629634e188b34c3b70fd2f631, 58c97a6afa064f2e8c45cd82d942fdf3; no voters:
I20260924 03:20:01.511199 409 raft_consensus.cc:2781] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:20:01.511370 409 raft_consensus.cc:482] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:20:01.511487 409 raft_consensus.cc:3037] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:01.513700 409 raft_consensus.cc:504] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.514297 409 leader_election.cc:288] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [CANDIDATE]: Term 1 election: Requested vote from peers 58c97a6afa064f2e8c45cd82d942fdf3 (127.14.227.131:36767), 8648e5461d8a4162b7c8bd1692bab8be (127.14.227.130:37083)
I20260924 03:20:01.514770 339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d7eb519359940e486441afbe8828b72" candidate_uuid: "5467410629634e188b34c3b70fd2f631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58c97a6afa064f2e8c45cd82d942fdf3"
I20260924 03:20:01.514889 32674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d7eb519359940e486441afbe8828b72" candidate_uuid: "5467410629634e188b34c3b70fd2f631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8648e5461d8a4162b7c8bd1692bab8be"
I20260924 03:20:01.515017 339 raft_consensus.cc:3037] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:01.515105 32674 raft_consensus.cc:3037] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:01.518357 32674 raft_consensus.cc:2454] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5467410629634e188b34c3b70fd2f631 in term 1.
I20260924 03:20:01.518352 339 raft_consensus.cc:2454] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5467410629634e188b34c3b70fd2f631 in term 1.
I20260924 03:20:01.518729 32473 leader_election.cc:302] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5467410629634e188b34c3b70fd2f631, 8648e5461d8a4162b7c8bd1692bab8be; no voters:
I20260924 03:20:01.519068 409 raft_consensus.cc:2781] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:01.519451 409 raft_consensus.cc:686] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 1 LEADER]: Becoming Leader. State: Replica: 5467410629634e188b34c3b70fd2f631, State: Running, Role: LEADER
I20260924 03:20:01.519941 409 consensus_queue.cc:237] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } }
I20260924 03:20:01.524704 32396 catalog_manager.cc:5495] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5467410629634e188b34c3b70fd2f631 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "5467410629634e188b34c3b70fd2f631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5467410629634e188b34c3b70fd2f631" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 33079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20260924 03:20:01.600469 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.595104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.14.227.129:33079)
0924 03:20:01.595288 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.595292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.595398 (+ 106us) server_negotiation.cc:407] Connection header received
0924 03:20:01.595570 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.595574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.595662 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.595793 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:01.596104 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.596803 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.598372 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.598831 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.598934 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.599411 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:01.599985 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:01.600123 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.600191 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.600254 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:20:01.608851 339 raft_consensus.cc:1261] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 1 FOLLOWER]: Refusing update from remote peer 5467410629634e188b34c3b70fd2f631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:01.608897 32674 raft_consensus.cc:1261] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 1 FOLLOWER]: Refusing update from remote peer 5467410629634e188b34c3b70fd2f631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:01.609807 413 consensus_queue.cc:1035] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8648e5461d8a4162b7c8bd1692bab8be" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:01.610232 409 consensus_queue.cc:1035] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58c97a6afa064f2e8c45cd82d942fdf3" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:01.623982 418 mvcc.cc:205] Tried to move back new op lower bound from 7332741126579589120 to 7332741126228992000. Current Snapshot: MvccSnapshot[applied={T|T < 7332741126579589120}]
I20260924 03:20:01.636909 32396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:01.637223 32396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:01.641587 32396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecea0b64e8fa49aa97e6b2f16fcb331c: Sending DeleteTablet for 3 replicas of tablet 5d7eb519359940e486441afbe8828b72
I20260924 03:20:01.643054 32654 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d7eb519359940e486441afbe8828b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:01 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:53492
I20260924 03:20:01.643419 319 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d7eb519359940e486441afbe8828b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:01 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:39728
I20260924 03:20:01.644083 32520 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d7eb519359940e486441afbe8828b72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:01 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:50102
I20260924 03:20:01.645617 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 32457
I20260924 03:20:01.666400 428 tablet_replica.cc:330] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: stopping tablet replica
I20260924 03:20:01.667006 428 raft_consensus.cc:2229] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:20:01.667953 428 raft_consensus.cc:2258] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:01.668852 427 tablet_replica.cc:330] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: stopping tablet replica
I20260924 03:20:01.669193 427 raft_consensus.cc:2229] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:01.669639 427 raft_consensus.cc:2258] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:01.671916 428 ts_tablet_manager.cc:1905] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:01.674484 426 tablet_replica.cc:330] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: stopping tablet replica
I20260924 03:20:01.674948 426 raft_consensus.cc:2229] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:01.677938 427 ts_tablet_manager.cc:1905] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:01.678321 426 raft_consensus.cc:2258] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:01.680769 426 ts_tablet_manager.cc:1905] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:01.684866 427 ts_tablet_manager.cc:1918] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:01.685024 427 log.cc:1196] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/5d7eb519359940e486441afbe8828b72
I20260924 03:20:01.685540 427 ts_tablet_manager.cc:1939] T 5d7eb519359940e486441afbe8828b72 P 58c97a6afa064f2e8c45cd82d942fdf3: Deleting consensus metadata
I20260924 03:20:01.686569 428 ts_tablet_manager.cc:1918] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:01.686759 428 log.cc:1196] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/5d7eb519359940e486441afbe8828b72
I20260924 03:20:01.687294 428 ts_tablet_manager.cc:1939] T 5d7eb519359940e486441afbe8828b72 P 5467410629634e188b34c3b70fd2f631: Deleting consensus metadata
I20260924 03:20:01.687549 32380 catalog_manager.cc:4841] TS 58c97a6afa064f2e8c45cd82d942fdf3 (127.14.227.131:36767): tablet 5d7eb519359940e486441afbe8828b72 (table test-table [id=d5b5cdd0e12c4566a33fbe83b3bf5c91]) successfully deleted
I20260924 03:20:01.688861 32383 catalog_manager.cc:4841] TS 5467410629634e188b34c3b70fd2f631 (127.14.227.129:33079): tablet 5d7eb519359940e486441afbe8828b72 (table test-table [id=d5b5cdd0e12c4566a33fbe83b3bf5c91]) successfully deleted
Sep 24 03:20:01 dist-test-slave-vd0p krb5kdc[32351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:20:01.693147 426 ts_tablet_manager.cc:1918] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:01.693318 426 log.cc:1196] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/5d7eb519359940e486441afbe8828b72
I20260924 03:20:01.693871 426 ts_tablet_manager.cc:1939] T 5d7eb519359940e486441afbe8828b72 P 8648e5461d8a4162b7c8bd1692bab8be: Deleting consensus metadata
I20260924 03:20:01.695853 32380 catalog_manager.cc:4841] TS 8648e5461d8a4162b7c8bd1692bab8be (127.14.227.130:37083): tablet 5d7eb519359940e486441afbe8828b72 (table test-table [id=d5b5cdd0e12c4566a33fbe83b3bf5c91]) successfully deleted
I20260924 03:20:01.698449 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.647073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.14.227.129:33079)
0924 03:20:01.647259 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.647265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.671314 (+ 24049us) server_negotiation.cc:407] Connection header received
0924 03:20:01.671864 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.671875 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.671976 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.672050 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:01.675553 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.676802 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.681814 (+ 5012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.682585 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.682693 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.691701 (+ 9008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:01.691712 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:01.691731 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:01.691770 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:01.695229 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.696156 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.696164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.696166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.696331 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.696741 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.696744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.696747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.697176 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:01.697503 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.697792 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.698069 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:20:01.699127 32580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:01.800447 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 32457
I20260924 03:20:01.818837 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 32591
Sep 24 03:20:01 dist-test-slave-vd0p krb5kdc[32351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:20:01.833068 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.819617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.14.227.130:37083)
0924 03:20:01.819803 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.819807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.819841 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:20:01.820008 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.820012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.820073 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.820196 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:01.821452 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.822225 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.823430 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.823806 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.823905 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.827968 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:01.827981 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:01.827996 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:01.828025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:01.830657 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.831290 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.831296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.831299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.831415 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.831709 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.831712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.831714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.832074 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:01.832286 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.832539 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.832694 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:20:01.833688 32714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:01.922405 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 32591
I20260924 03:20:01.938588 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 32724
Sep 24 03:20:01 dist-test-slave-vd0p krb5kdc[32351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790219999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:20:01.953631 398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:01.939345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.14.227.131:36767)
0924 03:20:01.939539 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:01.939545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:01.939565 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:20:01.939757 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:01.939762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:01.939849 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:01.939955 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:01.941291 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.942076 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.943372 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:01.943766 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:01.943921 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:01.948322 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:01.948336 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:01.948351 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:01.948383 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:01.950962 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.951584 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.951591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.951594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.951725 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:01.952112 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:01.952118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:01.952122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:01.952482 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:01.952776 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:01.953077 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 03:20:01.953305 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:20:01.954422 379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:02.042284 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 32724
I20260924 03:20:02.058794 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 32364
I20260924 03:20:02.060072 32426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:02.170393 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 32364
2026-09-24T03:20:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4381 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T03:20:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:02Z Disabled control of system clock
I20260924 03:20:02.249848 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46221
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:46221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:20:02.574872 437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:02.575188 437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:02.575270 437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:02.582671 437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:02.582784 437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:02.582823 437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:02.582873 437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:02.582919 437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:02.592127 437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:46221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:02.593384 437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:02.595158 437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:02.606709 442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:02.606662 443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:02.606797 445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:02.607642 437 server_base.cc:1034] running on GCE node
I20260924 03:20:02.608443 437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:02.609863 437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:02.611109 437 hybrid_clock.cc:648] HybridClock initialized: now 1790220002611067 us; error 33 us; skew 500 ppm
I20260924 03:20:02.611457 437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:02.617384 437 webserver.cc:460] Webserver started at http://127.14.227.190:42159/ using document root <none> and password file <none>
I20260924 03:20:02.618167 437 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:02.618299 437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:02.618615 437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:02.621572 437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "22357f5ec86a4045ad8ff03dd4a98c53"
format_stamp: "Formatted at 2026-09-24 03:20:02 on dist-test-slave-vd0p"
server_key: "e61e58aeef165a9e23654db5dcdc9906"
server_key_iv: "ce0d185dccc90019f2ce57978ee15e6f"
server_key_version: "encryptionkey@0"
I20260924 03:20:02.622407 437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "22357f5ec86a4045ad8ff03dd4a98c53"
format_stamp: "Formatted at 2026-09-24 03:20:02 on dist-test-slave-vd0p"
server_key: "e61e58aeef165a9e23654db5dcdc9906"
server_key_iv: "ce0d185dccc90019f2ce57978ee15e6f"
server_key_version: "encryptionkey@0"
I20260924 03:20:02.627677 437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:20:02.631331 451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:02.632771 437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:20:02.632975 437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "22357f5ec86a4045ad8ff03dd4a98c53"
format_stamp: "Formatted at 2026-09-24 03:20:02 on dist-test-slave-vd0p"
server_key: "e61e58aeef165a9e23654db5dcdc9906"
server_key_iv: "ce0d185dccc90019f2ce57978ee15e6f"
server_key_version: "encryptionkey@0"
I20260924 03:20:02.633199 437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:02.658295 437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:02.659179 437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:02.659485 437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:02.678252 437 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:46221
I20260924 03:20:02.678277 502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:46221 every 8 connection(s)
I20260924 03:20:02.679625 437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:02.682116 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 437
I20260924 03:20:02.682289 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:02.682734 15246 external_mini_cluster.cc:1428] Setting key cc347284c53c70b4094f679ff6f6b32c
I20260924 03:20:02.683748 503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:02.697117 506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:02.684632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.14.227.190:46221)
0924 03:20:02.685593 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:02.685604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:02.685644 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:20:02.688022 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:02.688044 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:02.688495 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:02.688812 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:02.689427 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:02.690867 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:02.692716 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:02.693365 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:02.693503 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:02.694785 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:02.694791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:02.694805 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:02.695381 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:02.695464 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:02.695487 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:02.695570 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:02.695796 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 03:20:02.696163 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":369,"threads_started":1}
I20260924 03:20:02.697366 503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53: Bootstrap starting.
I20260924 03:20:02.700583 503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:02.701701 503 log.cc:824] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:02.704378 503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53: No bootstrap required, opened a new log
I20260924 03:20:02.711896 503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22357f5ec86a4045ad8ff03dd4a98c53" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46221 } }
I20260924 03:20:02.712482 503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:02.712567 503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22357f5ec86a4045ad8ff03dd4a98c53, State: Initialized, Role: FOLLOWER
I20260924 03:20:02.713140 503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22357f5ec86a4045ad8ff03dd4a98c53" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46221 } }
I20260924 03:20:02.713344 503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:02.713449 503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:02.713569 503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:02.715771 503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22357f5ec86a4045ad8ff03dd4a98c53" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46221 } }
I20260924 03:20:02.716224 503 leader_election.cc:302] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22357f5ec86a4045ad8ff03dd4a98c53; no voters:
I20260924 03:20:02.716776 503 leader_election.cc:288] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:02.716924 508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:02.717583 508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [term 1 LEADER]: Becoming Leader. State: Replica: 22357f5ec86a4045ad8ff03dd4a98c53, State: Running, Role: LEADER
I20260924 03:20:02.718240 508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22357f5ec86a4045ad8ff03dd4a98c53" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46221 } }
I20260924 03:20:02.719273 503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:02.720932 510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22357f5ec86a4045ad8ff03dd4a98c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22357f5ec86a4045ad8ff03dd4a98c53" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46221 } } }
I20260924 03:20:02.721406 510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:02.721231 509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22357f5ec86a4045ad8ff03dd4a98c53. Latest consensus state: current_term: 1 leader_uuid: "22357f5ec86a4045ad8ff03dd4a98c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22357f5ec86a4045ad8ff03dd4a98c53" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46221 } } }
I20260924 03:20:02.722270 509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:02.723344 513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:02.728497 513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:02.736354 513 catalog_manager.cc:1347] Generated new cluster ID: c30161ce86a04ff9b1e8bcc9b5b903d2
I20260924 03:20:02.736476 513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:02.759467 513 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:02.761377 513 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:02.777549 513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53: Generated new TSK 0
I20260924 03:20:02.778620 513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:02.788782 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46221
--builtin_ntp_servers=127.14.227.148:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:20:03.109201 527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:03.109479 527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:03.109642 527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:03.116667 527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:03.116776 527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:03.116974 527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:20:03.126019 527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:03.127192 527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:03.128894 527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:03.141782 532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:03.141865 533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:03.141883 535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:03.142767 527 server_base.cc:1034] running on GCE node
I20260924 03:20:03.143369 527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:03.144244 527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:03.145457 527 hybrid_clock.cc:648] HybridClock initialized: now 1790220003145408 us; error 59 us; skew 500 ppm
I20260924 03:20:03.145797 527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:03.148077 527 webserver.cc:460] Webserver started at http://127.14.227.129:41635/ using document root <none> and password file <none>
I20260924 03:20:03.148720 527 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:03.148842 527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:03.149142 527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:03.151885 527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "33fb591f48b34415abf2da299baed97e"
format_stamp: "Formatted at 2026-09-24 03:20:03 on dist-test-slave-vd0p"
server_key: "090b0bf4e5c6001c5a03ddcabbd56711"
server_key_iv: "9b24423970ce2e40b25a2b196a3a299a"
server_key_version: "encryptionkey@0"
I20260924 03:20:03.152660 527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "33fb591f48b34415abf2da299baed97e"
format_stamp: "Formatted at 2026-09-24 03:20:03 on dist-test-slave-vd0p"
server_key: "090b0bf4e5c6001c5a03ddcabbd56711"
server_key_iv: "9b24423970ce2e40b25a2b196a3a299a"
server_key_version: "encryptionkey@0"
I20260924 03:20:03.157510 527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:20:03.160795 541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:03.162081 527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:03.162245 527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "33fb591f48b34415abf2da299baed97e"
format_stamp: "Formatted at 2026-09-24 03:20:03 on dist-test-slave-vd0p"
server_key: "090b0bf4e5c6001c5a03ddcabbd56711"
server_key_iv: "9b24423970ce2e40b25a2b196a3a299a"
server_key_version: "encryptionkey@0"
I20260924 03:20:03.162448 527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:03.201499 527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:03.202463 527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:03.202765 527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:03.203819 527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:03.205245 527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:03.205366 527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:03.205483 527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:03.205566 527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:03.238929 527 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:39697
I20260924 03:20:03.239022 653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:39697 every 8 connection(s)
I20260924 03:20:03.240245 527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:20:03.243223 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 527
I20260924 03:20:03.243422 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:20:03.244050 15246 external_mini_cluster.cc:1428] Setting key 232121decfec2a367029f7e091ff4d3b
I20260924 03:20:03.247037 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46221
--builtin_ntp_servers=127.14.227.148:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:20:03.254405 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:03.244263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:39069 (local address 127.14.227.190:46221)
0924 03:20:03.244871 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:03.244878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:03.245592 (+ 714us) server_negotiation.cc:407] Connection header received
0924 03:20:03.247701 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:03.247707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:03.247807 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:03.247947 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:03.249034 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:03.249777 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:03.251401 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:03.251834 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:03.252013 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:03.253579 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:03.253584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:03.253587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:03.253605 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:03.253648 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:03.253660 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:03.253736 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:03.254069 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 03:20:03.254155 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":296,"threads_started":1}
I20260924 03:20:03.256004 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:03.244170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46221 (local address 127.14.227.129:39069)
0924 03:20:03.245392 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:03.245469 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:03.247412 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:03.248178 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:03.248188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:03.248285 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:03.248858 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:03.248876 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:03.249967 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:03.249977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:03.251243 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:03.251256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:03.252100 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:03.252109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:03.252375 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:03.253167 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:03.253172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:03.253247 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:03.253360 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:03.253769 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:03.253782 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:03.253792 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 03:20:03.254056 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 03:20:03.254509 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":530,"threads_started":1}
I20260924 03:20:03.257609 654 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46221
I20260924 03:20:03.257907 654 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:03.258689 654 heartbeater.cc:511] Master 127.14.227.190:46221 requested a full tablet report, sending...
I20260924 03:20:03.260635 468 ts_manager.cc:194] Registered new tserver with Master: 33fb591f48b34415abf2da299baed97e (127.14.227.129:39697)
I20260924 03:20:03.262276 468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.129:39069
W20260924 03:20:03.579743 658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:03.580098 658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:03.580271 658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:03.587726 658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:03.587880 658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:03.588056 658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:20:03.597414 658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:03.598835 658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:03.600677 658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:03.614029 663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:03.613989 666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:03.614073 664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:03.619434 658 server_base.cc:1034] running on GCE node
I20260924 03:20:03.620160 658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:03.621199 658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:03.622426 658 hybrid_clock.cc:648] HybridClock initialized: now 1790220003622367 us; error 43 us; skew 500 ppm
I20260924 03:20:03.622758 658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:03.625133 658 webserver.cc:460] Webserver started at http://127.14.227.130:36123/ using document root <none> and password file <none>
I20260924 03:20:03.625818 658 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:03.625950 658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:03.626353 658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:03.629133 658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "5b9964b4771543238b92b5814b78259b"
format_stamp: "Formatted at 2026-09-24 03:20:03 on dist-test-slave-vd0p"
server_key: "68d725758528ed80f6e3597cab5abd19"
server_key_iv: "0b8c227d4d574d05533575a0a327d4c6"
server_key_version: "encryptionkey@0"
I20260924 03:20:03.629915 658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "5b9964b4771543238b92b5814b78259b"
format_stamp: "Formatted at 2026-09-24 03:20:03 on dist-test-slave-vd0p"
server_key: "68d725758528ed80f6e3597cab5abd19"
server_key_iv: "0b8c227d4d574d05533575a0a327d4c6"
server_key_version: "encryptionkey@0"
I20260924 03:20:03.635129 658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:20:03.638720 672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:03.640012 658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:20:03.640196 658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "5b9964b4771543238b92b5814b78259b"
format_stamp: "Formatted at 2026-09-24 03:20:03 on dist-test-slave-vd0p"
server_key: "68d725758528ed80f6e3597cab5abd19"
server_key_iv: "0b8c227d4d574d05533575a0a327d4c6"
server_key_version: "encryptionkey@0"
I20260924 03:20:03.640444 658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:03.663911 658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:03.664732 658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:03.665066 658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:03.666179 658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:03.667711 658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:03.667853 658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:03.667984 658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:03.668114 658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:03.703893 658 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:38627
I20260924 03:20:03.703982 784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:38627 every 8 connection(s)
I20260924 03:20:03.705300 658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:20:03.712291 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 658
I20260924 03:20:03.712466 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:20:03.713019 15246 external_mini_cluster.cc:1428] Setting key 42fd0f5faf02c7aadcc9735681709733
I20260924 03:20:03.716277 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46221
--builtin_ntp_servers=127.14.227.148:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:20:03.719468 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:03.707967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:55277 (local address 127.14.227.190:46221)
0924 03:20:03.708149 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:03.708153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:03.709574 (+ 1421us) server_negotiation.cc:407] Connection header received
0924 03:20:03.711449 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:03.711455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:03.711549 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:03.711659 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:03.713115 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:03.714127 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:03.716209 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:03.716797 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:03.716901 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:03.718642 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:03.718646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:03.718649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:03.718675 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:03.718708 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:03.718720 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:03.718791 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:03.719111 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 03:20:03.719193 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:20:03.720901 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:03.708540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46221 (local address 127.14.227.130:55277)
0924 03:20:03.709414 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:03.709449 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:03.711147 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:03.711952 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:03.711964 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:03.712098 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:03.712889 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:03.712915 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:03.714282 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:03.714287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:03.716033 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:03.716052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:03.717028 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:03.717037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:03.717398 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:03.718261 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:03.718267 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:03.718321 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:03.718430 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:03.718821 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:03.718834 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:03.718845 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:20:03.719067 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 03:20:03.719509 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":262,"threads_started":1}
I20260924 03:20:03.722395 785 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46221
I20260924 03:20:03.722730 785 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:03.723640 785 heartbeater.cc:511] Master 127.14.227.190:46221 requested a full tablet report, sending...
I20260924 03:20:03.725293 468 ts_manager.cc:194] Registered new tserver with Master: 5b9964b4771543238b92b5814b78259b (127.14.227.130:38627)
I20260924 03:20:03.726186 468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.130:55277
W20260924 03:20:04.036393 788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:04.036804 788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:04.036974 788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:04.045732 788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:04.045814 788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:04.046034 788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:20:04.054641 788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:04.055876 788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:04.057562 788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:04.070416 793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:04.070384 796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:04.070456 794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:04.071041 788 server_base.cc:1034] running on GCE node
I20260924 03:20:04.071594 788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:04.072487 788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:04.073694 788 hybrid_clock.cc:648] HybridClock initialized: now 1790220004073646 us; error 38 us; skew 500 ppm
I20260924 03:20:04.074059 788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:04.076226 788 webserver.cc:460] Webserver started at http://127.14.227.131:39179/ using document root <none> and password file <none>
I20260924 03:20:04.076924 788 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:04.077035 788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:04.077346 788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:04.080060 788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "ec865054ce7c4bd3a4f64352425f4399"
format_stamp: "Formatted at 2026-09-24 03:20:04 on dist-test-slave-vd0p"
server_key: "d0366daaad72b1bbefc0d81ed9c61681"
server_key_iv: "1b9f626f1bb9023e637dbf9005f3bd32"
server_key_version: "encryptionkey@0"
I20260924 03:20:04.080786 788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "ec865054ce7c4bd3a4f64352425f4399"
format_stamp: "Formatted at 2026-09-24 03:20:04 on dist-test-slave-vd0p"
server_key: "d0366daaad72b1bbefc0d81ed9c61681"
server_key_iv: "1b9f626f1bb9023e637dbf9005f3bd32"
server_key_version: "encryptionkey@0"
I20260924 03:20:04.085489 788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 03:20:04.088601 802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:04.089876 788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:20:04.090067 788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "ec865054ce7c4bd3a4f64352425f4399"
format_stamp: "Formatted at 2026-09-24 03:20:04 on dist-test-slave-vd0p"
server_key: "d0366daaad72b1bbefc0d81ed9c61681"
server_key_iv: "1b9f626f1bb9023e637dbf9005f3bd32"
server_key_version: "encryptionkey@0"
I20260924 03:20:04.090269 788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:04.112046 788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:04.112792 788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:04.113091 788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:04.113983 788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:04.115300 788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:04.115447 788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:04.115581 788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:04.115664 788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:04.148128 788 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:36043
I20260924 03:20:04.148193 914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:36043 every 8 connection(s)
I20260924 03:20:04.149400 788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:20:04.152516 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 788
I20260924 03:20:04.152717 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:20:04.153286 15246 external_mini_cluster.cc:1428] Setting key fa1c478087589b91c5eaf234f3ec3cab
I20260924 03:20:04.161651 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.151872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:47921 (local address 127.14.227.190:46221)
0924 03:20:04.152045 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.152049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.153668 (+ 1619us) server_negotiation.cc:407] Connection header received
0924 03:20:04.155463 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.155469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.155564 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.155680 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:04.156719 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.157450 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.159065 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.159455 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.159592 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.160826 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:04.160831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:04.160834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:04.160850 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:04.160881 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:04.160892 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:04.160980 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.161241 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.161335 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:20:04.162657 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.152440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46221 (local address 127.14.227.131:47921)
0924 03:20:04.153507 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.153550 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:04.155206 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:04.155899 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:04.155908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:04.155999 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:04.156552 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.156567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.157605 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.157614 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.158892 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.158904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.159687 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.159695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.159922 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.160524 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:04.160527 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:04.160562 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:04.160634 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:04.160994 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:04.161009 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:04.161020 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:20:04.161231 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 03:20:04.161587 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":372,"threads_started":1}
I20260924 03:20:04.163954 915 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46221
I20260924 03:20:04.164268 915 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:04.164958 915 heartbeater.cc:511] Master 127.14.227.190:46221 requested a full tablet report, sending...
I20260924 03:20:04.166360 468 ts_manager.cc:194] Registered new tserver with Master: ec865054ce7c4bd3a4f64352425f4399 (127.14.227.131:36043)
I20260924 03:20:04.167434 468 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.131:47921
I20260924 03:20:04.168233 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:20:04.177760 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.172418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.14.227.190:46221)
0924 03:20:04.172564 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.172577 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.173044 (+ 467us) server_negotiation.cc:407] Connection header received
0924 03:20:04.173173 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.173176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.173241 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.173323 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:04.173755 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.174504 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.175773 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.176158 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.176251 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.177242 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:04.177247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:04.177250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:04.177265 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:04.177293 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:04.177304 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:04.177369 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.177484 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.177548 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:20:04.181277 467 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:20:04.187963 467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:20:04.218470 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.200446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.14.227.131:36043)
0924 03:20:04.210110 (+ 9664us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.210119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.210155 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:20:04.210250 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.210259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.210571 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.210766 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:04.211532 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.213217 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.215670 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.216996 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.217192 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.217403 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.217867 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.218071 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9429,"thread_start_us":9250,"threads_started":1}
I20260924 03:20:04.218459 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.199772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:36043 (local address 127.0.0.1:59912)
0924 03:20:04.209732 (+ 9960us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.209759 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:04.209873 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:04.210850 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:04.210879 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:04.210918 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:04.211330 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.211345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.213369 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.213375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.215476 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.215488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.217506 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.217510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.217654 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.217686 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:20:04.217823 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 03:20:04.218039 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9769,"mutex_wait_us":80,"thread_start_us":10821,"threads_started":1}
I20260924 03:20:04.219605 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.199640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:38627 (local address 127.0.0.1:55952)
0924 03:20:04.200419 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.200455 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:04.200618 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:04.212255 (+ 11637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:04.212259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:04.212288 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:04.212806 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.212817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.214694 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.214699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.216860 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.216876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.218751 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.218757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.218950 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.219015 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 03:20:04.219095 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:20:04.219252 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":20,"thread_start_us":373,"threads_started":1}
I20260924 03:20:04.220628 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.201394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.0.0.1:48198)
0924 03:20:04.210897 (+ 9503us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.210918 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:04.211182 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:04.212151 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:04.212156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:04.212179 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:04.212683 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.212699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.214157 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.214165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.216378 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.216394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.218731 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.218740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.220010 (+ 1270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.220046 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:20:04.220184 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 03:20:04.220311 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9337,"mutex_wait_us":3,"thread_start_us":8029,"threads_started":1}
I20260924 03:20:04.220759 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.202076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.14.227.129:39697)
0924 03:20:04.210404 (+ 8328us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.210416 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.211040 (+ 624us) server_negotiation.cc:407] Connection header received
0924 03:20:04.211380 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.211390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.211678 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.211871 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:04.212875 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.214028 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.216589 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.218150 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.218303 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.218519 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.220278 (+ 1759us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.220423 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8101,"thread_start_us":7968,"threads_started":1}
I20260924 03:20:04.222061 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.199836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.14.227.130:38627)
0924 03:20:04.211461 (+ 11625us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.211468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.211496 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:20:04.211580 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.211586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.211771 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.211955 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:04.213004 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.214464 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.217086 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.218529 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.218937 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.219151 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.219269 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.219430 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11455,"thread_start_us":11339,"threads_started":1}
I20260924 03:20:04.223027 850 tablet_service.cc:1467] Processing CreateTablet for tablet 07e098f885e947379858d65ff9909631 (DEFAULT_TABLE table=test-table [id=9eca9228deb84f5ebc72f4f14f1549a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:04.224486 850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e098f885e947379858d65ff9909631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:04.224217 589 tablet_service.cc:1467] Processing CreateTablet for tablet 07e098f885e947379858d65ff9909631 (DEFAULT_TABLE table=test-table [id=9eca9228deb84f5ebc72f4f14f1549a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:04.225477 589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e098f885e947379858d65ff9909631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:04.225414 720 tablet_service.cc:1467] Processing CreateTablet for tablet 07e098f885e947379858d65ff9909631 (DEFAULT_TABLE table=test-table [id=9eca9228deb84f5ebc72f4f14f1549a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:04.226670 720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e098f885e947379858d65ff9909631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:04.234669 933 tablet_bootstrap.cc:492] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Bootstrap starting.
I20260924 03:20:04.237855 933 tablet_bootstrap.cc:654] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:04.239190 933 log.cc:824] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:04.250142 935 tablet_bootstrap.cc:492] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Bootstrap starting.
I20260924 03:20:04.253017 933 tablet_bootstrap.cc:492] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: No bootstrap required, opened a new log
I20260924 03:20:04.253283 933 ts_tablet_manager.cc:1397] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20260924 03:20:04.254729 936 tablet_bootstrap.cc:492] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Bootstrap starting.
I20260924 03:20:04.258071 935 tablet_bootstrap.cc:654] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:04.259346 935 log.cc:824] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:04.260881 936 tablet_bootstrap.cc:654] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:04.261745 936 log.cc:824] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:04.262375 935 tablet_bootstrap.cc:492] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: No bootstrap required, opened a new log
I20260924 03:20:04.262629 935 ts_tablet_manager.cc:1397] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260924 03:20:04.263507 933 raft_consensus.cc:348] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.263976 933 raft_consensus.cc:374] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:04.264075 933 raft_consensus.cc:729] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec865054ce7c4bd3a4f64352425f4399, State: Initialized, Role: FOLLOWER
I20260924 03:20:04.265087 654 heartbeater.cc:503] Master 127.14.227.190:46221 was elected leader, sending a full tablet report...
I20260924 03:20:04.265853 936 tablet_bootstrap.cc:492] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: No bootstrap required, opened a new log
I20260924 03:20:04.266193 936 ts_tablet_manager.cc:1397] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 03:20:04.264765 933 consensus_queue.cc:260] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.273887 936 raft_consensus.cc:348] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.274348 936 raft_consensus.cc:374] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:04.274441 936 raft_consensus.cc:729] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fb591f48b34415abf2da299baed97e, State: Initialized, Role: FOLLOWER
I20260924 03:20:04.274782 915 heartbeater.cc:503] Master 127.14.227.190:46221 was elected leader, sending a full tablet report...
I20260924 03:20:04.274722 935 raft_consensus.cc:348] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.275123 935 raft_consensus.cc:374] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:04.275213 935 raft_consensus.cc:729] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9964b4771543238b92b5814b78259b, State: Initialized, Role: FOLLOWER
I20260924 03:20:04.275333 933 ts_tablet_manager.cc:1428] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20260924 03:20:04.275089 936 consensus_queue.cc:260] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.275771 935 consensus_queue.cc:260] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.277578 785 heartbeater.cc:503] Master 127.14.227.190:46221 was elected leader, sending a full tablet report...
I20260924 03:20:04.277685 936 ts_tablet_manager.cc:1428] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 03:20:04.278810 935 ts_tablet_manager.cc:1428] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260924 03:20:04.401239 916 tablet.cc:2369] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:20:04.457468 786 tablet.cc:2369] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:04.479763 939 raft_consensus.cc:482] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:20:04.480070 939 raft_consensus.cc:504] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.481824 939 leader_election.cc:288] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b9964b4771543238b92b5814b78259b (127.14.227.130:38627), 33fb591f48b34415abf2da299baed97e (127.14.227.129:39697)
I20260924 03:20:04.487335 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.482201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:38627 (local address 127.14.227.131:52359)
0924 03:20:04.482546 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.482566 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:04.482696 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:04.483030 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:04.483034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:04.483070 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:04.483280 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.483289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.484278 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.484285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.485548 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.485562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.486651 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.486657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.486830 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.486853 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:20:04.486955 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 03:20:04.487080 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260924 03:20:04.487345 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.482494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:52359 (local address 127.14.227.130:38627)
0924 03:20:04.482691 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.482696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.482716 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:20:04.482831 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.482837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.482907 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.483004 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:04.483438 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.484149 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.485698 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.486500 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.486662 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.486731 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.486980 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.487098 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:20:04.488664 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.482365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:52871 (local address 127.14.227.129:39697)
0924 03:20:04.482603 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.482613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.483307 (+ 694us) server_negotiation.cc:407] Connection header received
0924 03:20:04.483526 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.483530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.483622 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.483728 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:04.484351 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.485009 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.486784 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.487755 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.487916 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.487998 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.488349 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.488451 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:20:04.488726 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.482343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.14.227.131:52871)
0924 03:20:04.483189 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.483205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:04.483377 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:04.483741 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:04.483745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:04.483769 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:04.484151 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.484160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.485176 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.485185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.486588 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:04.486602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.487964 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.487970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:04.488171 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.488200 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 03:20:04.488303 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:20:04.488425 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":450,"threads_started":1}
I20260924 03:20:04.488528 740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e098f885e947379858d65ff9909631" candidate_uuid: "ec865054ce7c4bd3a4f64352425f4399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b9964b4771543238b92b5814b78259b" is_pre_election: true
I20260924 03:20:04.489121 740 raft_consensus.cc:2454] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec865054ce7c4bd3a4f64352425f4399 in term 0.
I20260924 03:20:04.489763 806 leader_election.cc:302] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9964b4771543238b92b5814b78259b, ec865054ce7c4bd3a4f64352425f4399; no voters:
I20260924 03:20:04.489854 609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e098f885e947379858d65ff9909631" candidate_uuid: "ec865054ce7c4bd3a4f64352425f4399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fb591f48b34415abf2da299baed97e" is_pre_election: true
I20260924 03:20:04.490221 939 raft_consensus.cc:2781] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:20:04.490382 939 raft_consensus.cc:482] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:20:04.490499 939 raft_consensus.cc:3037] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:04.490463 609 raft_consensus.cc:2454] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec865054ce7c4bd3a4f64352425f4399 in term 0.
W20260924 03:20:04.493085 655 tablet.cc:2369] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:04.493063 939 raft_consensus.cc:504] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.493805 939 leader_election.cc:288] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [CANDIDATE]: Term 1 election: Requested vote from peers 5b9964b4771543238b92b5814b78259b (127.14.227.130:38627), 33fb591f48b34415abf2da299baed97e (127.14.227.129:39697)
I20260924 03:20:04.494225 740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e098f885e947379858d65ff9909631" candidate_uuid: "ec865054ce7c4bd3a4f64352425f4399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b9964b4771543238b92b5814b78259b"
I20260924 03:20:04.494350 609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e098f885e947379858d65ff9909631" candidate_uuid: "ec865054ce7c4bd3a4f64352425f4399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fb591f48b34415abf2da299baed97e"
I20260924 03:20:04.494453 740 raft_consensus.cc:3037] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:04.494613 609 raft_consensus.cc:3037] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:04.496731 740 raft_consensus.cc:2454] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec865054ce7c4bd3a4f64352425f4399 in term 1.
I20260924 03:20:04.496954 609 raft_consensus.cc:2454] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec865054ce7c4bd3a4f64352425f4399 in term 1.
I20260924 03:20:04.497188 806 leader_election.cc:302] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9964b4771543238b92b5814b78259b, ec865054ce7c4bd3a4f64352425f4399; no voters:
I20260924 03:20:04.497491 939 raft_consensus.cc:2781] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:04.498035 939 raft_consensus.cc:686] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 1 LEADER]: Becoming Leader. State: Replica: ec865054ce7c4bd3a4f64352425f4399, State: Running, Role: LEADER
I20260924 03:20:04.498562 939 consensus_queue.cc:237] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } }
I20260924 03:20:04.503299 466 catalog_manager.cc:5495] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec865054ce7c4bd3a4f64352425f4399 (127.14.227.131). New cstate: current_term: 1 leader_uuid: "ec865054ce7c4bd3a4f64352425f4399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec865054ce7c4bd3a4f64352425f4399" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 36043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 } health_report { overall_health: UNKNOWN } } }
I20260924 03:20:04.588732 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.583110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.14.227.131:36043)
0924 03:20:04.583309 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.583315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.583440 (+ 125us) server_negotiation.cc:407] Connection header received
0924 03:20:04.583705 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.583709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.583794 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.583944 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:04.584308 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.585079 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.586646 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.587050 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.587177 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.587658 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:04.588184 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:04.588350 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.588404 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.588480 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:20:04.596930 609 raft_consensus.cc:1261] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 1 FOLLOWER]: Refusing update from remote peer ec865054ce7c4bd3a4f64352425f4399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:04.596931 740 raft_consensus.cc:1261] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 1 FOLLOWER]: Refusing update from remote peer ec865054ce7c4bd3a4f64352425f4399: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:04.598069 943 consensus_queue.cc:1035] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33fb591f48b34415abf2da299baed97e" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:04.598459 939 consensus_queue.cc:1035] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b9964b4771543238b92b5814b78259b" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 38627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:04.626075 467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:04.626391 467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:04.629863 467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22357f5ec86a4045ad8ff03dd4a98c53: Sending DeleteTablet for 3 replicas of tablet 07e098f885e947379858d65ff9909631
I20260924 03:20:04.630926 720 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e098f885e947379858d65ff9909631 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:04 UTC) from {username='slave'} at 127.0.0.1:55952
I20260924 03:20:04.631023 589 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e098f885e947379858d65ff9909631 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:04 UTC) from {username='slave'} at 127.0.0.1:48198
I20260924 03:20:04.631675 850 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e098f885e947379858d65ff9909631 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:04 UTC) from {username='slave'} at 127.0.0.1:59912
I20260924 03:20:04.632810 958 tablet_replica.cc:330] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: stopping tablet replica
I20260924 03:20:04.633031 957 tablet_replica.cc:330] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: stopping tablet replica
I20260924 03:20:04.633206 956 tablet_replica.cc:330] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: stopping tablet replica
I20260924 03:20:04.633231 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 527
I20260924 03:20:04.633231 958 raft_consensus.cc:2229] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:20:04.633785 956 raft_consensus.cc:2229] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:04.633983 958 raft_consensus.cc:2258] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:04.634850 957 raft_consensus.cc:2229] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:04.635444 956 raft_consensus.cc:2258] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:04.635963 957 raft_consensus.cc:2258] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:04.638072 956 ts_tablet_manager.cc:1905] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:04.638329 958 ts_tablet_manager.cc:1905] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:04.639349 957 ts_tablet_manager.cc:1905] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:04.644066 956 ts_tablet_manager.cc:1918] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:04.644255 956 log.cc:1196] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/07e098f885e947379858d65ff9909631
I20260924 03:20:04.644858 956 ts_tablet_manager.cc:1939] T 07e098f885e947379858d65ff9909631 P 5b9964b4771543238b92b5814b78259b: Deleting consensus metadata
I20260924 03:20:04.645401 958 ts_tablet_manager.cc:1918] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:04.645588 958 log.cc:1196] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/07e098f885e947379858d65ff9909631
I20260924 03:20:04.646226 958 ts_tablet_manager.cc:1939] T 07e098f885e947379858d65ff9909631 P ec865054ce7c4bd3a4f64352425f4399: Deleting consensus metadata
I20260924 03:20:04.646884 455 catalog_manager.cc:4841] TS 5b9964b4771543238b92b5814b78259b (127.14.227.130:38627): tablet 07e098f885e947379858d65ff9909631 (table test-table [id=9eca9228deb84f5ebc72f4f14f1549a1]) successfully deleted
I20260924 03:20:04.647840 454 catalog_manager.cc:4841] TS ec865054ce7c4bd3a4f64352425f4399 (127.14.227.131:36043): tablet 07e098f885e947379858d65ff9909631 (table test-table [id=9eca9228deb84f5ebc72f4f14f1549a1]) successfully deleted
I20260924 03:20:04.649847 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.634516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.14.227.129:39697)
0924 03:20:04.634706 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.634713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.636833 (+ 2120us) server_negotiation.cc:407] Connection header received
0924 03:20:04.637187 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.637193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.637298 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.637391 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:04.638219 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.639299 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.645486 (+ 6187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.646211 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.646335 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.648427 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:04.648436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:04.648448 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:04.649033 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:04.649080 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:04.649099 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:04.649223 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.649369 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.649472 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:20:04.650677 649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:04.650674 957 ts_tablet_manager.cc:1918] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:04.650920 957 log.cc:1196] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/07e098f885e947379858d65ff9909631
I20260924 03:20:04.780597 957 ts_tablet_manager.cc:1939] T 07e098f885e947379858d65ff9909631 P 33fb591f48b34415abf2da299baed97e: Deleting consensus metadata
I20260924 03:20:04.780784 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 527
W20260924 03:20:04.797325 452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.227.129:39697 (error 108)
I20260924 03:20:04.798477 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 658
I20260924 03:20:04.799461 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.798582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.0.0.1:48224)
0924 03:20:04.799076 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.799247 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
W20260924 03:20:04.799774 452 catalog_manager.cc:4584] TS 33fb591f48b34415abf2da299baed97e (127.14.227.129:39697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07e098f885e947379858d65ff9909631: Network error: Client connection negotiation failed: client connection to 127.14.227.129:39697: connect: Connection refused (error 111)
I20260924 03:20:04.805470 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.799270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.14.227.130:38627)
0924 03:20:04.799458 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.799465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.799512 (+ 47us) server_negotiation.cc:407] Connection header received
0924 03:20:04.799752 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.799758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.799855 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.799964 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:04.800384 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.801104 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.802463 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.802902 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.803038 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.804114 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:04.804123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:04.804137 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:04.804708 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:04.804753 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:04.804771 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:04.804892 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.805017 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.805118 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 03:20:04.806234 780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:04.853657 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.853126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.0.0.1:48232)
0924 03:20:04.853374 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.853484 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 03:20:04.854835 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.854404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.0.0.1:48238)
0924 03:20:04.854580 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.854667 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 03:20:04.894543 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 658
I20260924 03:20:04.911386 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 788
I20260924 03:20:04.915228 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.914734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.0.0.1:48252)
0924 03:20:04.914938 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.915046 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 03:20:04.916540 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.916125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.0.0.1:48254)
0924 03:20:04.916290 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.916386 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 03:20:04.919275 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.912242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.14.227.131:36043)
0924 03:20:04.912449 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:04.912458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:04.912485 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:20:04.912615 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:04.912621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:04.912745 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:04.912863 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:04.913241 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.914413 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.916039 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:04.916639 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:04.916813 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:04.917874 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:04.917882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:04.917895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:04.918535 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:04.918581 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:04.918600 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:04.918723 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:04.918872 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 03:20:04.918972 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:20:04.920084 910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:04.997571 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.997000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.0.0.1:48262)
0924 03:20:04.997269 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.997399 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 03:20:04.998837 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:04.998380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39697 (local address 127.0.0.1:48266)
0924 03:20:04.998602 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:04.998690 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 03:20:05.038338 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 788
I20260924 03:20:05.057061 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 437
I20260924 03:20:05.058203 498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:05.166468 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 437
2026-09-24T03:20:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2990 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:05 dist-test-slave-vd0p krb5kdc[968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:05 dist-test-slave-vd0p krb5kdc[968](info): set up 2 sockets
Sep 24 03:20:05 dist-test-slave-vd0p krb5kdc[968](info): commencing operation
krb5kdc: starting...
W20260924 03:20:06.261878 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:06 dist-test-slave-vd0p krb5kdc[968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:06.495792 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:45277
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:06.818379 981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:06.818657 981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:06.818851 981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:06.825784 981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:06.825882 981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:06.825996 981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:06.826109 981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:06.826195 981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:06.835640 981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:45277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:06.836951 981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:06.838303 981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:06.850152 986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:06.850122 987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:06.850186 989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:06.850980 981 server_base.cc:1034] running on GCE node
I20260924 03:20:06.851812 981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:06.853214 981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:06.854410 981 hybrid_clock.cc:648] HybridClock initialized: now 1790220006854346 us; error 52 us; skew 500 ppm
I20260924 03:20:06.854755 981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:06 dist-test-slave-vd0p krb5kdc[968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220006, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:06.857793 981 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:06.859347 981 webserver.cc:460] Webserver started at http://127.14.227.190:33657/ using document root <none> and password file <none>
I20260924 03:20:06.860074 981 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:06.860203 981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:06.860510 981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:06.863453 981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "157cc57e41d54d89a98adbc03ac332a0"
format_stamp: "Formatted at 2026-09-24 03:20:06 on dist-test-slave-vd0p"
server_key: "a7a6f495a7a615a5afc99492417b4a9e"
server_key_iv: "5ae16fcd9d51bfdeabb4231ce7543642"
server_key_version: "encryptionkey@0"
I20260924 03:20:06.864200 981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "157cc57e41d54d89a98adbc03ac332a0"
format_stamp: "Formatted at 2026-09-24 03:20:06 on dist-test-slave-vd0p"
server_key: "a7a6f495a7a615a5afc99492417b4a9e"
server_key_iv: "5ae16fcd9d51bfdeabb4231ce7543642"
server_key_version: "encryptionkey@0"
I20260924 03:20:06.869305 981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:20:06.872576 996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:06.873796 981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:20:06.873991 981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "157cc57e41d54d89a98adbc03ac332a0"
format_stamp: "Formatted at 2026-09-24 03:20:06 on dist-test-slave-vd0p"
server_key: "a7a6f495a7a615a5afc99492417b4a9e"
server_key_iv: "5ae16fcd9d51bfdeabb4231ce7543642"
server_key_version: "encryptionkey@0"
I20260924 03:20:06.874183 981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:06.898747 981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:06.902697 981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:06.903008 981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:06.921316 981 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:45277
I20260924 03:20:06.921340 1047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:45277 every 8 connection(s)
I20260924 03:20:06.922658 981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:06.926213 1048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:06.928702 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 981
I20260924 03:20:06.928879 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:06.929436 15246 external_mini_cluster.cc:1428] Setting key 8d8cdebf8d8c3f8f85e3beb86b5160b4
I20260924 03:20:06.938215 1048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0: Bootstrap starting.
I20260924 03:20:06.941798 1048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:06.943121 1048 log.cc:824] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:20:06 dist-test-slave-vd0p krb5kdc[968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:06.945832 1048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0: No bootstrap required, opened a new log
I20260924 03:20:06.953429 1051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:06.931342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.14.227.190:45277)
0924 03:20:06.932123 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:06.932134 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:06.932172 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:20:06.933756 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:06.933776 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:06.934196 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:06.934516 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:06.936250 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:06.937500 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:06.939304 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:06.939925 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:06.940040 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:06.945179 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:06.945186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:06.945199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:06.945231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:06.949170 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:06.949890 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:06.949900 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:06.949908 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:06.950122 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:06.950477 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:06.950482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:06.950484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:06.951116 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:06.951409 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:06.951713 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 03:20:06.952149 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":393,"threads_started":1}
I20260924 03:20:06.955920 1048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157cc57e41d54d89a98adbc03ac332a0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45277 } }
I20260924 03:20:06.956231 1048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:06.956300 1048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157cc57e41d54d89a98adbc03ac332a0, State: Initialized, Role: FOLLOWER
I20260924 03:20:06.956749 1048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157cc57e41d54d89a98adbc03ac332a0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45277 } }
I20260924 03:20:06.956940 1048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:06.957028 1048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:06.957147 1048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:06.959199 1048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157cc57e41d54d89a98adbc03ac332a0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45277 } }
I20260924 03:20:06.959610 1048 leader_election.cc:302] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 157cc57e41d54d89a98adbc03ac332a0; no voters:
I20260924 03:20:06.960093 1048 leader_election.cc:288] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:06.960278 1053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:06.960960 1053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [term 1 LEADER]: Becoming Leader. State: Replica: 157cc57e41d54d89a98adbc03ac332a0, State: Running, Role: LEADER
I20260924 03:20:06.961534 1053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157cc57e41d54d89a98adbc03ac332a0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45277 } }
I20260924 03:20:06.962108 1048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:06.964252 1055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 157cc57e41d54d89a98adbc03ac332a0. Latest consensus state: current_term: 1 leader_uuid: "157cc57e41d54d89a98adbc03ac332a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157cc57e41d54d89a98adbc03ac332a0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45277 } } }
I20260924 03:20:06.964550 1054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "157cc57e41d54d89a98adbc03ac332a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157cc57e41d54d89a98adbc03ac332a0" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45277 } } }
I20260924 03:20:06.964717 1055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:06.966089 1054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:06.967291 1061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:06.972456 1061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:06.981464 1061 catalog_manager.cc:1347] Generated new cluster ID: 8a2fc22ae00a4382afedfd1bca690d66
I20260924 03:20:06.981580 1061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:06.998129 1061 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:06.999682 1061 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:07.007754 1061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0: Generated new TSK 0
I20260924 03:20:07.008818 1061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:20:07.073520 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45277
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:07.408146 1074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:07.408463 1074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:07.408635 1074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:07.416167 1074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:07.416308 1074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:07.416491 1074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:20:07.426174 1074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:07.427407 1074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:07.428725 1074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:07.442024 1079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:07.442003 1080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:07.442117 1082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:07.442848 1074 server_base.cc:1034] running on GCE node
I20260924 03:20:07.443485 1074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:07.444407 1074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:07.445641 1074 hybrid_clock.cc:648] HybridClock initialized: now 1790220007445584 us; error 54 us; skew 500 ppm
I20260924 03:20:07.445936 1074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:07 dist-test-slave-vd0p krb5kdc[968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220007, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:07.448982 1074 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:20:07.450330 1074 webserver.cc:460] Webserver started at http://127.14.227.129:42729/ using document root <none> and password file <none>
I20260924 03:20:07.450960 1074 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:07.451056 1074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:07.451295 1074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:07.454258 1074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4"
format_stamp: "Formatted at 2026-09-24 03:20:07 on dist-test-slave-vd0p"
server_key: "88af91f96912f5650fbb7f79822927e8"
server_key_iv: "e4f8543ea814d70b2d27d17c4b0dad66"
server_key_version: "encryptionkey@0"
I20260924 03:20:07.454936 1074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4"
format_stamp: "Formatted at 2026-09-24 03:20:07 on dist-test-slave-vd0p"
server_key: "88af91f96912f5650fbb7f79822927e8"
server_key_iv: "e4f8543ea814d70b2d27d17c4b0dad66"
server_key_version: "encryptionkey@0"
I20260924 03:20:07.459811 1074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:20:07.463043 1089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:07.464395 1074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:07.464555 1074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4"
format_stamp: "Formatted at 2026-09-24 03:20:07 on dist-test-slave-vd0p"
server_key: "88af91f96912f5650fbb7f79822927e8"
server_key_iv: "e4f8543ea814d70b2d27d17c4b0dad66"
server_key_version: "encryptionkey@0"
I20260924 03:20:07.464725 1074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:07.506222 1074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:07.510699 1074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:07.511001 1074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:07.511978 1074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:07.513324 1074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:07.513394 1074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:07.513535 1074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:07.513599 1074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:07.546063 1074 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:43311
I20260924 03:20:07.546235 1201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:43311 every 8 connection(s)
I20260924 03:20:07.547326 1074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:20:07.548971 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 1074
I20260924 03:20:07.549178 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:20:07.549819 15246 external_mini_cluster.cc:1428] Setting key a285bbd34338df4f25915553a8030dc2
Sep 24 03:20:07 dist-test-slave-vd0p krb5kdc[968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220007, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:07.568624 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:07.550276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:49341 (local address 127.14.227.190:45277)
0924 03:20:07.550751 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:07.550757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:07.551730 (+ 973us) server_negotiation.cc:407] Connection header received
0924 03:20:07.553529 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:07.553534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:07.553622 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:07.553740 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:07.555195 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:07.555867 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:07.557349 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:07.557737 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:07.557863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:07.561408 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:07.561417 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:07.561424 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:07.561466 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:07.565137 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:07.566000 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:07.566008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:07.566014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:07.566152 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:07.566631 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:07.566639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:07.566645 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:07.567014 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:07.567162 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:07.567923 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 03:20:07.568162 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":225,"threads_started":1}
I20260924 03:20:07.569497 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:07.550792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45277 (local address 127.14.227.129:49341)
0924 03:20:07.551575 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:07.551622 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:07.553282 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:07.554004 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:07.554015 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:07.554509 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:07.555028 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:07.555042 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:07.556018 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:07.556023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:07.557172 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:07.557188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:07.557988 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:07.557997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:07.558227 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:07.558886 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:07.558907 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:07.561189 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:07.565287 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:07.565293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:07.565305 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:07.565831 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:07.566283 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:07.566287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:07.566289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:07.566497 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:07.567146 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:07.567151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:07.567498 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 03:20:07.567802 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 03:20:07.568291 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":338,"threads_started":1}
I20260924 03:20:07.571187 1202 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45277
I20260924 03:20:07.571497 1202 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:07.572259 1202 heartbeater.cc:511] Master 127.14.227.190:45277 requested a full tablet report, sending...
I20260924 03:20:07.574663 1013 ts_manager.cc:194] Registered new tserver with Master: d6d0954d4d794de8bcfb71e6e2d7f2e4 (127.14.227.129:43311)
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:20:07.576226 1013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:49341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:20:07.604533 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45277
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:07.915715 1208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:07.916028 1208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:07.916185 1208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:07.923153 1208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:07.923332 1208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:07.923504 1208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:20:07.932574 1208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:07.933748 1208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:07.934965 1208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:07.947216 1214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:07.947280 1213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:07.947281 1216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:07.947974 1208 server_base.cc:1034] running on GCE node
I20260924 03:20:07.948490 1208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:07.949278 1208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:07.950484 1208 hybrid_clock.cc:648] HybridClock initialized: now 1790220007950481 us; error 112 us; skew 500 ppm
I20260924 03:20:07.950759 1208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:07 dist-test-slave-vd0p krb5kdc[968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220007, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:07.958880 1208 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:20:07.960301 1208 webserver.cc:460] Webserver started at http://127.14.227.130:41541/ using document root <none> and password file <none>
I20260924 03:20:07.960927 1208 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:07.961064 1208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:07.961375 1208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:07.964140 1208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "5c2f12f00ee8487eb0cae7ab4fb69158"
format_stamp: "Formatted at 2026-09-24 03:20:07 on dist-test-slave-vd0p"
server_key: "93ce84e356cca2eb1a25ba8a269e09b7"
server_key_iv: "cdb08d05b82fcf4f7f74ffc377804193"
server_key_version: "encryptionkey@0"
I20260924 03:20:07.964870 1208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "5c2f12f00ee8487eb0cae7ab4fb69158"
format_stamp: "Formatted at 2026-09-24 03:20:07 on dist-test-slave-vd0p"
server_key: "93ce84e356cca2eb1a25ba8a269e09b7"
server_key_iv: "cdb08d05b82fcf4f7f74ffc377804193"
server_key_version: "encryptionkey@0"
I20260924 03:20:07.969858 1208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:20:07.973033 1223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:07.974421 1208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:07.974593 1208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "5c2f12f00ee8487eb0cae7ab4fb69158"
format_stamp: "Formatted at 2026-09-24 03:20:07 on dist-test-slave-vd0p"
server_key: "93ce84e356cca2eb1a25ba8a269e09b7"
server_key_iv: "cdb08d05b82fcf4f7f74ffc377804193"
server_key_version: "encryptionkey@0"
I20260924 03:20:07.974776 1208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:08.015170 1208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:08.019109 1208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:08.019410 1208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:08.020344 1208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:08.021610 1208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:08.021709 1208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:08.021857 1208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:08.021930 1208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:08.054590 1208 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:39369
I20260924 03:20:08.054641 1335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:39369 every 8 connection(s)
I20260924 03:20:08.055816 1208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:20:08.057557 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 1208
I20260924 03:20:08.057776 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:20:08.058346 15246 external_mini_cluster.cc:1428] Setting key b9e4aec97ce688c1300f90a00cb4239d
Sep 24 03:20:08 dist-test-slave-vd0p krb5kdc[968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220007, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:08.077483 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.058319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:50511 (local address 127.14.227.190:45277)
0924 03:20:08.058550 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.058562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.059884 (+ 1322us) server_negotiation.cc:407] Connection header received
0924 03:20:08.061748 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.061754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.061843 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.061969 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:08.063575 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.064687 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.066157 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.066930 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.067094 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.070840 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:08.070849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:08.070857 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:08.070902 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:08.074365 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.075050 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.075058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.075064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.075211 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.075587 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.075595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.075601 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.075960 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:08.076114 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.076725 (+ 611us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.076971 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 03:20:08.078123 1338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.058871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45277 (local address 127.14.227.130:50511)
0924 03:20:08.059719 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:08.059762 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:08.061482 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:08.062226 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:08.062234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:08.062785 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:08.063420 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.063439 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.064845 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.064849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.066040 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.066051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.067151 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.067158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.067346 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.068070 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:08.068103 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:08.070644 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:08.074491 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:08.074497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:08.074521 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:08.074919 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:08.075311 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:08.075314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:08.075316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:08.075469 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:08.076077 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:08.076082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:08.076368 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 03:20:08.076571 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 03:20:08.076917 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":354,"threads_started":1}
I20260924 03:20:08.079524 1336 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45277
I20260924 03:20:08.079861 1336 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:08.080691 1336 heartbeater.cc:511] Master 127.14.227.190:45277 requested a full tablet report, sending...
I20260924 03:20:08.082206 1013 ts_manager.cc:194] Registered new tserver with Master: 5c2f12f00ee8487eb0cae7ab4fb69158 (127.14.227.130:39369)
I20260924 03:20:08.082957 1013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:50511
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:20:08.114892 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45277
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:08.423686 1341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:08.423947 1341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:08.424118 1341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:08.430974 1341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:08.431072 1341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:08.431285 1341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:20:08.440583 1341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:08.442270 1341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:08.443982 1341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:08.459956 1349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:08.459986 1347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:08.460333 1346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:08.460738 1341 server_base.cc:1034] running on GCE node
I20260924 03:20:08.461534 1341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:08.462708 1341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:08.463956 1341 hybrid_clock.cc:648] HybridClock initialized: now 1790220008463884 us; error 79 us; skew 500 ppm
I20260924 03:20:08.464324 1341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:08 dist-test-slave-vd0p krb5kdc[968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220008, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:08.468009 1341 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:20:08.469761 1341 webserver.cc:460] Webserver started at http://127.14.227.131:43505/ using document root <none> and password file <none>
I20260924 03:20:08.470635 1341 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:08.470779 1341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:08.471100 1341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:08.475690 1341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "66ef23670f6d475782626a7a6ac53ba1"
format_stamp: "Formatted at 2026-09-24 03:20:08 on dist-test-slave-vd0p"
server_key: "c9783edbc8fe19bab9624e2bb6c8770f"
server_key_iv: "0975eb418a7389ec999f6412b998251b"
server_key_version: "encryptionkey@0"
I20260924 03:20:08.476678 1341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "66ef23670f6d475782626a7a6ac53ba1"
format_stamp: "Formatted at 2026-09-24 03:20:08 on dist-test-slave-vd0p"
server_key: "c9783edbc8fe19bab9624e2bb6c8770f"
server_key_iv: "0975eb418a7389ec999f6412b998251b"
server_key_version: "encryptionkey@0"
I20260924 03:20:08.482489 1341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 03:20:08.486156 1356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:08.487912 1341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:20:08.488065 1341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "66ef23670f6d475782626a7a6ac53ba1"
format_stamp: "Formatted at 2026-09-24 03:20:08 on dist-test-slave-vd0p"
server_key: "c9783edbc8fe19bab9624e2bb6c8770f"
server_key_iv: "0975eb418a7389ec999f6412b998251b"
server_key_version: "encryptionkey@0"
I20260924 03:20:08.488232 1341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:08.508807 1341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:08.513085 1341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:08.513324 1341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:08.514256 1341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:08.515666 1341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:08.515741 1341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:08.515825 1341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:08.515870 1341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:08.547923 1341 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:39057
I20260924 03:20:08.547968 1468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:39057 every 8 connection(s)
I20260924 03:20:08.549093 1341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:20:08.555650 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 1341
I20260924 03:20:08.555855 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:20:08.556274 15246 external_mini_cluster.cc:1428] Setting key e35214f1e2d43390934864019ce25d25
Sep 24 03:20:08 dist-test-slave-vd0p krb5kdc[968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220008, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:08.568069 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.551737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:42349 (local address 127.14.227.190:45277)
0924 03:20:08.551902 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.551906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.552888 (+ 982us) server_negotiation.cc:407] Connection header received
0924 03:20:08.554760 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.554767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.554848 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.554965 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:08.556430 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.557381 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.559154 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.559544 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.559637 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.562909 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:08.562913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:08.562916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:08.562946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:08.565453 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.566146 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.566150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.566152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.566270 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.566645 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.566649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.566651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.566885 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:08.566992 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.567642 (+ 650us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.567833 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:20:08.568988 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.552080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45277 (local address 127.14.227.131:42349)
0924 03:20:08.552742 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:08.552774 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:08.554446 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:08.555172 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:08.555180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:08.555633 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:08.556242 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.556259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.557554 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.557558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.559045 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.559055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.559821 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.559827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.560006 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.560585 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:08.560610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:08.562705 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:08.565565 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:08.565570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:08.565580 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:08.566047 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:08.566359 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:08.566362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:08.566364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:08.566524 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:08.566990 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:08.566995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:08.567350 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 03:20:08.567555 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 03:20:08.567897 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":236,"threads_started":1}
I20260924 03:20:08.570362 1469 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45277
I20260924 03:20:08.570663 1469 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:08.571331 1469 heartbeater.cc:511] Master 127.14.227.190:45277 requested a full tablet report, sending...
I20260924 03:20:08.572683 1012 ts_manager.cc:194] Registered new tserver with Master: 66ef23670f6d475782626a7a6ac53ba1 (127.14.227.131:39057)
I20260924 03:20:08.573455 1012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:42349
I20260924 03:20:08.579857 1202 heartbeater.cc:503] Master 127.14.227.190:45277 was elected leader, sending a full tablet report...
I20260924 03:20:08.583106 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:20:08.599099 1204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.587048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.14.227.190:45277)
0924 03:20:08.587213 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.587217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.587648 (+ 431us) server_negotiation.cc:407] Connection header received
0924 03:20:08.587794 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.587798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.587851 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.587931 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:08.589163 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.589840 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.591047 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.591472 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.591573 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.594207 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:08.594211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:08.594215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:08.594245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:08.596623 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.597293 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.597296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.597299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.597415 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.597816 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.597819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.597821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.598177 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:08.598363 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.598732 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.598879 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:20:08.602780 1012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:20:08.609423 1012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:20:08.636193 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.621409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.14.227.131:39057)
0924 03:20:08.623209 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.623221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.625019 (+ 1798us) server_negotiation.cc:407] Connection header received
0924 03:20:08.625138 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.625148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.625460 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.625676 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:08.626980 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.628974 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.632349 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.634237 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.634479 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.634718 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.635702 (+ 984us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.635869 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1602,"thread_start_us":1427,"threads_started":1}
I20260924 03:20:08.636193 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.620782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:39057 (local address 127.0.0.1:50368)
0924 03:20:08.624621 (+ 3839us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:08.624654 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:08.624799 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:08.626055 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:08.626066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:08.626105 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:08.626717 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.626734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.629165 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.629174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.632128 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.632147 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.635259 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.635265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.635476 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.635512 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:20:08.635649 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 03:20:08.635820 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3655,"thread_start_us":2629,"threads_started":1}
I20260924 03:20:08.637939 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.620552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:39369 (local address 127.0.0.1:60670)
0924 03:20:08.623800 (+ 3248us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:08.623843 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:08.624033 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:08.626472 (+ 2439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:08.626479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:08.626518 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:08.627069 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.627083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.631005 (+ 3922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.631013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.635289 (+ 4276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.635303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.637127 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.637135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.637281 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.637311 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:20:08.637397 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:20:08.637555 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2982,"mutex_wait_us":1,"thread_start_us":14343,"threads_started":1}
I20260924 03:20:08.638739 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.620790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.14.227.130:39369)
0924 03:20:08.625712 (+ 4922us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.625724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.625817 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:20:08.625926 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.625936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.626227 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.627307 (+ 1080us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:08.627570 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.630828 (+ 3258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.635509 (+ 4681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.636914 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.637093 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.637257 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.638291 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.638466 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4761,"thread_start_us":4657,"threads_started":1}
I20260924 03:20:08.640205 1404 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8432d20292484aa2d43cd051543c91 (DEFAULT_TABLE table=test-table [id=1df1be3654a642469d374a2c2fabf895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:08.641723 1404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8432d20292484aa2d43cd051543c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:08.642005 1271 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8432d20292484aa2d43cd051543c91 (DEFAULT_TABLE table=test-table [id=1df1be3654a642469d374a2c2fabf895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:08.643277 1271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8432d20292484aa2d43cd051543c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:08.643329 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.623709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.14.227.129:43311)
0924 03:20:08.635423 (+ 11714us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.635432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.635458 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:20:08.635551 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.635562 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.635793 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.635991 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:08.637112 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.639686 (+ 2574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.641216 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.642223 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.642429 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.642586 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.642843 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.642981 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11510,"thread_start_us":12441,"threads_started":1}
I20260924 03:20:08.643234 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.623051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:43311 (local address 127.0.0.1:56126)
0924 03:20:08.635162 (+ 12111us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:08.635183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:08.635295 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:08.636613 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:08.636618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:08.636647 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:08.636934 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.636947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.639826 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.639831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.641079 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.641090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.642427 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.642433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.642679 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.642701 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:20:08.642810 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:20:08.642923 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11970,"thread_start_us":1218,"threads_started":1}
I20260924 03:20:08.646641 1137 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8432d20292484aa2d43cd051543c91 (DEFAULT_TABLE table=test-table [id=1df1be3654a642469d374a2c2fabf895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:08.647789 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8432d20292484aa2d43cd051543c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:08.656024 1487 tablet_bootstrap.cc:492] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: Bootstrap starting.
I20260924 03:20:08.658499 1488 tablet_bootstrap.cc:492] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: Bootstrap starting.
I20260924 03:20:08.659325 1489 tablet_bootstrap.cc:492] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: Bootstrap starting.
I20260924 03:20:08.661047 1487 tablet_bootstrap.cc:654] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:08.661821 1488 tablet_bootstrap.cc:654] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:08.662204 1487 log.cc:824] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:08.662611 1489 tablet_bootstrap.cc:654] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:08.663069 1488 log.cc:824] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:08.663769 1489 log.cc:824] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:08.665397 1487 tablet_bootstrap.cc:492] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: No bootstrap required, opened a new log
I20260924 03:20:08.665640 1487 ts_tablet_manager.cc:1397] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 03:20:08.666522 1489 tablet_bootstrap.cc:492] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: No bootstrap required, opened a new log
I20260924 03:20:08.666522 1488 tablet_bootstrap.cc:492] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: No bootstrap required, opened a new log
I20260924 03:20:08.666762 1488 ts_tablet_manager.cc:1397] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260924 03:20:08.666762 1489 ts_tablet_manager.cc:1397] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 03:20:08.673853 1487 raft_consensus.cc:348] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.674263 1487 raft_consensus.cc:374] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:08.674373 1487 raft_consensus.cc:729] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef23670f6d475782626a7a6ac53ba1, State: Initialized, Role: FOLLOWER
I20260924 03:20:08.674973 1487 consensus_queue.cc:260] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.676002 1469 heartbeater.cc:503] Master 127.14.227.190:45277 was elected leader, sending a full tablet report...
I20260924 03:20:08.676648 1487 ts_tablet_manager.cc:1428] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 03:20:08.677150 1489 raft_consensus.cc:348] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.677152 1488 raft_consensus.cc:348] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.677554 1489 raft_consensus.cc:374] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:08.677574 1488 raft_consensus.cc:374] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:08.677659 1489 raft_consensus.cc:729] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d0954d4d794de8bcfb71e6e2d7f2e4, State: Initialized, Role: FOLLOWER
I20260924 03:20:08.677659 1488 raft_consensus.cc:729] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2f12f00ee8487eb0cae7ab4fb69158, State: Initialized, Role: FOLLOWER
I20260924 03:20:08.678244 1489 consensus_queue.cc:260] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.678244 1488 consensus_queue.cc:260] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.680171 1489 ts_tablet_manager.cc:1428] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 03:20:08.680256 1488 ts_tablet_manager.cc:1428] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 03:20:08.680846 1336 heartbeater.cc:503] Master 127.14.227.190:45277 was elected leader, sending a full tablet report...
I20260924 03:20:08.701510 1494 raft_consensus.cc:482] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:20:08.701745 1494 raft_consensus.cc:504] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.703351 1494 leader_election.cc:288] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c2f12f00ee8487eb0cae7ab4fb69158 (127.14.227.130:39369), 66ef23670f6d475782626a7a6ac53ba1 (127.14.227.131:39057)
I20260924 03:20:08.710467 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.703922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:47195 (local address 127.14.227.131:39057)
0924 03:20:08.704233 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.704242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.704741 (+ 499us) server_negotiation.cc:407] Connection header received
0924 03:20:08.704911 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.704919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.705018 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.705120 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:08.705794 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.706684 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.708533 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.709394 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.709520 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.709588 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.710044 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.710195 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 03:20:08.710529 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.703732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:39057 (local address 127.14.227.129:47195)
0924 03:20:08.704582 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:08.704605 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:08.704758 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:08.705152 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:08.705157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:08.705184 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:08.705616 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.705630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.706818 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.706823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.708359 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.708370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.709608 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.709613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.709809 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.709909 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 03:20:08.709996 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:20:08.710081 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":54,"thread_start_us":426,"threads_started":1}
I20260924 03:20:08.710577 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.703957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:57517 (local address 127.14.227.130:39369)
0924 03:20:08.704412 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.704421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.705029 (+ 608us) server_negotiation.cc:407] Connection header received
0924 03:20:08.705199 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.705205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.705288 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.705390 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:08.706052 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.706814 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.708780 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.709593 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.709752 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.709817 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.710203 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.710332 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313}
I20260924 03:20:08.710740 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.703923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:39369 (local address 127.14.227.129:57517)
0924 03:20:08.704906 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:08.704925 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:08.705060 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:08.705420 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:08.705423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:08.705443 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:08.705868 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.705881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.706977 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.706981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.708626 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:08.708641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.709828 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.709833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:08.710023 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.710057 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:20:08.710155 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 03:20:08.710288 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":2,"thread_start_us":593,"threads_started":1}
I20260924 03:20:08.711544 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8432d20292484aa2d43cd051543c91" candidate_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef23670f6d475782626a7a6ac53ba1" is_pre_election: true
I20260924 03:20:08.711720 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8432d20292484aa2d43cd051543c91" candidate_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" is_pre_election: true
I20260924 03:20:08.712016 1424 raft_consensus.cc:2454] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6d0954d4d794de8bcfb71e6e2d7f2e4 in term 0.
I20260924 03:20:08.712294 1291 raft_consensus.cc:2454] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6d0954d4d794de8bcfb71e6e2d7f2e4 in term 0.
I20260924 03:20:08.712725 1093 leader_election.cc:302] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef23670f6d475782626a7a6ac53ba1, d6d0954d4d794de8bcfb71e6e2d7f2e4; no voters:
I20260924 03:20:08.713165 1494 raft_consensus.cc:2781] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:20:08.713315 1494 raft_consensus.cc:482] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:20:08.713451 1494 raft_consensus.cc:3037] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:08.715596 1494 raft_consensus.cc:504] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.716125 1494 leader_election.cc:288] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [CANDIDATE]: Term 1 election: Requested vote from peers 5c2f12f00ee8487eb0cae7ab4fb69158 (127.14.227.130:39369), 66ef23670f6d475782626a7a6ac53ba1 (127.14.227.131:39057)
I20260924 03:20:08.716549 1291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8432d20292484aa2d43cd051543c91" candidate_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158"
I20260924 03:20:08.716563 1424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8432d20292484aa2d43cd051543c91" candidate_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef23670f6d475782626a7a6ac53ba1"
I20260924 03:20:08.716748 1424 raft_consensus.cc:3037] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:08.716753 1291 raft_consensus.cc:3037] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:08.719031 1424 raft_consensus.cc:2454] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6d0954d4d794de8bcfb71e6e2d7f2e4 in term 1.
I20260924 03:20:08.719198 1291 raft_consensus.cc:2454] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6d0954d4d794de8bcfb71e6e2d7f2e4 in term 1.
I20260924 03:20:08.719467 1093 leader_election.cc:302] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef23670f6d475782626a7a6ac53ba1, d6d0954d4d794de8bcfb71e6e2d7f2e4; no voters:
I20260924 03:20:08.719750 1494 raft_consensus.cc:2781] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:08.720293 1494 raft_consensus.cc:686] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 1 LEADER]: Becoming Leader. State: Replica: d6d0954d4d794de8bcfb71e6e2d7f2e4, State: Running, Role: LEADER
I20260924 03:20:08.720757 1494 consensus_queue.cc:237] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } }
I20260924 03:20:08.725754 1011 catalog_manager.cc:5495] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6d0954d4d794de8bcfb71e6e2d7f2e4 (127.14.227.129). New cstate: current_term: 1 leader_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6d0954d4d794de8bcfb71e6e2d7f2e4" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 43311 } health_report { overall_health: HEALTHY } } }
I20260924 03:20:08.756968 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.751300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.14.227.129:43311)
0924 03:20:08.751513 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.751520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.751623 (+ 103us) server_negotiation.cc:407] Connection header received
0924 03:20:08.751863 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.751868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.751958 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.752118 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:08.752531 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.753226 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.754655 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.755083 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.755225 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.755817 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:08.756326 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:08.756518 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.756675 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.756724 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:20:08.765453 1291 raft_consensus.cc:1261] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 1 FOLLOWER]: Refusing update from remote peer d6d0954d4d794de8bcfb71e6e2d7f2e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:08.765447 1424 raft_consensus.cc:1261] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 1 FOLLOWER]: Refusing update from remote peer d6d0954d4d794de8bcfb71e6e2d7f2e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:08.766549 1498 consensus_queue.cc:1035] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef23670f6d475782626a7a6ac53ba1" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 39057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:08.766981 1494 consensus_queue.cc:1035] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c2f12f00ee8487eb0cae7ab4fb69158" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:08.777801 1501 mvcc.cc:205] Tried to move back new op lower bound from 7332741155892404224 to 7332741155723669504. Current Snapshot: MvccSnapshot[applied={T|T < 7332741155892404224}]
I20260924 03:20:08.779958 1503 mvcc.cc:205] Tried to move back new op lower bound from 7332741155892404224 to 7332741155723669504. Current Snapshot: MvccSnapshot[applied={T|T < 7332741155892404224}]
I20260924 03:20:08.784214 1502 mvcc.cc:205] Tried to move back new op lower bound from 7332741155892404224 to 7332741155723669504. Current Snapshot: MvccSnapshot[applied={T|T < 7332741155892404224}]
I20260924 03:20:08.793493 1011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:08.793780 1011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:08.797572 1011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 157cc57e41d54d89a98adbc03ac332a0: Sending DeleteTablet for 3 replicas of tablet 3b8432d20292484aa2d43cd051543c91
I20260924 03:20:08.798859 1271 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8432d20292484aa2d43cd051543c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:08 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:60670
I20260924 03:20:08.799052 1404 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8432d20292484aa2d43cd051543c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:08 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:50368
I20260924 03:20:08.799283 1137 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8432d20292484aa2d43cd051543c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:08 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:56126
I20260924 03:20:08.805461 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 1074
I20260924 03:20:08.810362 1512 tablet_replica.cc:330] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: stopping tablet replica
I20260924 03:20:08.811069 1512 raft_consensus.cc:2229] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:20:08.812073 1512 raft_consensus.cc:2258] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:08.812752 1511 tablet_replica.cc:330] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: stopping tablet replica
I20260924 03:20:08.813110 1511 raft_consensus.cc:2229] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:08.813589 1511 raft_consensus.cc:2258] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:08.815706 1512 ts_tablet_manager.cc:1905] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:08.816596 1510 tablet_replica.cc:330] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: stopping tablet replica
I20260924 03:20:08.816984 1510 raft_consensus.cc:2229] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:08.817564 1510 raft_consensus.cc:2258] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:08.819483 1510 ts_tablet_manager.cc:1905] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:08.823848 1512 ts_tablet_manager.cc:1918] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:08.824067 1512 log.cc:1196] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/3b8432d20292484aa2d43cd051543c91
I20260924 03:20:08.824684 1512 ts_tablet_manager.cc:1939] T 3b8432d20292484aa2d43cd051543c91 P d6d0954d4d794de8bcfb71e6e2d7f2e4: Deleting consensus metadata
I20260924 03:20:08.826297 1510 ts_tablet_manager.cc:1918] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:08.826459 1510 log.cc:1196] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/3b8432d20292484aa2d43cd051543c91
I20260924 03:20:08.826985 1510 ts_tablet_manager.cc:1939] T 3b8432d20292484aa2d43cd051543c91 P 5c2f12f00ee8487eb0cae7ab4fb69158: Deleting consensus metadata
I20260924 03:20:08.827384 1511 ts_tablet_manager.cc:1905] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:20:08 dist-test-slave-vd0p krb5kdc[968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:20:08.828761 999 catalog_manager.cc:4841] TS 5c2f12f00ee8487eb0cae7ab4fb69158 (127.14.227.130:39369): tablet 3b8432d20292484aa2d43cd051543c91 (table test-table [id=1df1be3654a642469d374a2c2fabf895]) successfully deleted
I20260924 03:20:08.830646 998 catalog_manager.cc:4841] TS d6d0954d4d794de8bcfb71e6e2d7f2e4 (127.14.227.129:43311): tablet 3b8432d20292484aa2d43cd051543c91 (table test-table [id=1df1be3654a642469d374a2c2fabf895]) successfully deleted
I20260924 03:20:08.834995 1511 ts_tablet_manager.cc:1918] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:08.835208 1511 log.cc:1196] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/3b8432d20292484aa2d43cd051543c91
I20260924 03:20:08.835812 1511 ts_tablet_manager.cc:1939] T 3b8432d20292484aa2d43cd051543c91 P 66ef23670f6d475782626a7a6ac53ba1: Deleting consensus metadata
I20260924 03:20:08.837395 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.806693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.14.227.129:43311)
0924 03:20:08.806854 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.806861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.815831 (+ 8970us) server_negotiation.cc:407] Connection header received
0924 03:20:08.816027 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.816033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.816124 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.816252 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:08.818498 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.819749 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.821853 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.822734 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.822835 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.829242 (+ 6407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:08.829256 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:08.829275 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:08.829318 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:08.833647 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.834628 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.834639 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.834646 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.834857 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.835373 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.835382 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.835386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.835934 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:08.836320 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.836680 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.836939 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:20:08.837877 1000 catalog_manager.cc:4841] TS 66ef23670f6d475782626a7a6ac53ba1 (127.14.227.131:39057): tablet 3b8432d20292484aa2d43cd051543c91 (table test-table [id=1df1be3654a642469d374a2c2fabf895]) successfully deleted
I20260924 03:20:08.838208 1197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:08.938264 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 1074
I20260924 03:20:08.952384 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 1208
Sep 24 03:20:08 dist-test-slave-vd0p krb5kdc[968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:20:08.967905 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:08.953153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.14.227.130:39369)
0924 03:20:08.953351 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:08.953358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:08.953378 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:20:08.953572 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:08.953576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:08.953638 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:08.953740 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:08.955090 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.955829 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.957117 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:08.957507 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:08.957654 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:08.962214 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:08.962229 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:08.962252 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:08.962291 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:08.965170 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.965810 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.965826 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.965829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.966216 (+ 387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:08.966671 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:08.966674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:08.966676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:08.967026 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:08.967254 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:08.967497 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 03:20:08.967643 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:20:08.968556 1331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:09.060521 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 1208
I20260924 03:20:09.078536 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 1341
Sep 24 03:20:09 dist-test-slave-vd0p krb5kdc[968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:20:09.093897 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:09.079304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.14.227.131:39057)
0924 03:20:09.079491 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:09.079495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:09.079515 (+ 20us) server_negotiation.cc:407] Connection header received
0924 03:20:09.079680 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:09.079683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:09.079757 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:09.079845 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:09.081648 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:09.082445 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:09.083656 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:09.084030 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:09.084118 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:09.088252 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:09.088265 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:09.088281 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:09.088312 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:09.091323 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:09.091994 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:09.092000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:09.092003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:09.092138 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:09.092561 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:09.092564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:09.092566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:09.092933 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:09.093138 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:09.093375 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 03:20:09.093553 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:20:09.094549 1464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:09.184538 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 1341
I20260924 03:20:09.200666 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 981
I20260924 03:20:09.201717 1043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:09.311556 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 981
2026-09-24T03:20:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4145 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T03:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:09Z Disabled control of system clock
I20260924 03:20:09.383858 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46413
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:46413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:20:09.706880 1521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:09.707173 1521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:09.707306 1521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:09.714708 1521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:09.714810 1521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:09.714895 1521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:09.715008 1521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:09.715092 1521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:09.724838 1521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:46413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:46413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:09.726155 1521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:09.727927 1521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:09.739414 1526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:09.739480 1529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:09.739559 1527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:09.740942 1521 server_base.cc:1034] running on GCE node
I20260924 03:20:09.741642 1521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:09.743041 1521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:09.744232 1521 hybrid_clock.cc:648] HybridClock initialized: now 1790220009744173 us; error 60 us; skew 500 ppm
I20260924 03:20:09.744570 1521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:09.747089 1521 webserver.cc:460] Webserver started at http://127.14.227.190:36007/ using document root <none> and password file <none>
I20260924 03:20:09.747757 1521 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:09.747869 1521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:09.748200 1521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:09.751279 1521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "ac590be141e643cf9866e489cd1b0f4f"
format_stamp: "Formatted at 2026-09-24 03:20:09 on dist-test-slave-vd0p"
server_key: "84ab3d8cc07deb6304e07258bc7a36c6"
server_key_iv: "6221311b5962a8c5ab332c69825a5695"
server_key_version: "encryptionkey@0"
I20260924 03:20:09.752094 1521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "ac590be141e643cf9866e489cd1b0f4f"
format_stamp: "Formatted at 2026-09-24 03:20:09 on dist-test-slave-vd0p"
server_key: "84ab3d8cc07deb6304e07258bc7a36c6"
server_key_iv: "6221311b5962a8c5ab332c69825a5695"
server_key_version: "encryptionkey@0"
I20260924 03:20:09.757175 1521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 03:20:09.760833 1535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:09.762228 1521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:09.762434 1521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "ac590be141e643cf9866e489cd1b0f4f"
format_stamp: "Formatted at 2026-09-24 03:20:09 on dist-test-slave-vd0p"
server_key: "84ab3d8cc07deb6304e07258bc7a36c6"
server_key_iv: "6221311b5962a8c5ab332c69825a5695"
server_key_version: "encryptionkey@0"
I20260924 03:20:09.762657 1521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:09.780318 1521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:09.781141 1521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:09.781426 1521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:09.799439 1521 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:46413
I20260924 03:20:09.799453 1586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:46413 every 8 connection(s)
I20260924 03:20:09.800761 1521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:09.804471 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:09.807042 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 1521
I20260924 03:20:09.807207 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:09.807732 15246 external_mini_cluster.cc:1428] Setting key ae8117a6ea57c1492eca587296501cec
I20260924 03:20:09.816534 1587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f: Bootstrap starting.
I20260924 03:20:09.820129 1587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:09.821105 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:09.809600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.14.227.190:46413)
0924 03:20:09.810654 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:09.810664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:09.810703 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:20:09.812282 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:09.812304 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:09.812727 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:09.813051 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:09.813603 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:09.814957 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:09.816827 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:09.817482 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:09.817632 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:09.818859 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:09.818866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:09.818879 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:09.819397 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:09.819483 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:09.819503 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:09.819594 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:09.819809 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 03:20:09.820159 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":433,"threads_started":1}
I20260924 03:20:09.821151 1587 log.cc:824] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:09.823956 1587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f: No bootstrap required, opened a new log
I20260924 03:20:09.830906 1587 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac590be141e643cf9866e489cd1b0f4f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46413 } }
I20260924 03:20:09.831296 1587 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:09.831405 1587 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac590be141e643cf9866e489cd1b0f4f, State: Initialized, Role: FOLLOWER
I20260924 03:20:09.832000 1587 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac590be141e643cf9866e489cd1b0f4f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46413 } }
I20260924 03:20:09.832243 1587 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:09.832381 1587 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:09.832566 1587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:09.834723 1587 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac590be141e643cf9866e489cd1b0f4f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46413 } }
I20260924 03:20:09.835219 1587 leader_election.cc:302] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac590be141e643cf9866e489cd1b0f4f; no voters:
I20260924 03:20:09.835794 1587 leader_election.cc:288] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:09.835944 1592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:09.837059 1592 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [term 1 LEADER]: Becoming Leader. State: Replica: ac590be141e643cf9866e489cd1b0f4f, State: Running, Role: LEADER
I20260924 03:20:09.837615 1592 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac590be141e643cf9866e489cd1b0f4f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46413 } }
I20260924 03:20:09.838114 1587 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:09.840440 1594 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac590be141e643cf9866e489cd1b0f4f. Latest consensus state: current_term: 1 leader_uuid: "ac590be141e643cf9866e489cd1b0f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac590be141e643cf9866e489cd1b0f4f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46413 } } }
I20260924 03:20:09.840415 1593 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac590be141e643cf9866e489cd1b0f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac590be141e643cf9866e489cd1b0f4f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 46413 } } }
I20260924 03:20:09.840888 1594 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:09.840909 1593 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:09.841794 1600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:09.847369 1600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:09.855513 1600 catalog_manager.cc:1347] Generated new cluster ID: df6c1ea4ab95499781830b48458ba6ac
I20260924 03:20:09.855633 1600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:09.879431 1600 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:09.880856 1600 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:09.888216 1600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f: Generated new TSK 0
I20260924 03:20:09.889050 1600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:09.893110 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46413
--builtin_ntp_servers=127.14.227.148:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:20:10.243451 1611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:10.243799 1611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:10.243887 1611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:10.251483 1611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:10.251590 1611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:10.251698 1611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:20:10.261328 1611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:10.262648 1611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:10.264364 1611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:10.278772 1617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:10.278744 1619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:10.278717 1616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:10.279440 1611 server_base.cc:1034] running on GCE node
I20260924 03:20:10.280174 1611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:10.281155 1611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:10.282394 1611 hybrid_clock.cc:648] HybridClock initialized: now 1790220010282359 us; error 49 us; skew 500 ppm
I20260924 03:20:10.282733 1611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:10.285071 1611 webserver.cc:460] Webserver started at http://127.14.227.129:34219/ using document root <none> and password file <none>
I20260924 03:20:10.285797 1611 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:10.285938 1611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:10.286316 1611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:10.289153 1611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "1a852df683db4628b3a2491d26680b98"
format_stamp: "Formatted at 2026-09-24 03:20:10 on dist-test-slave-vd0p"
server_key: "027267bfef244ec84aa8cde3ae1adbcc"
server_key_iv: "03147fc8aa34d6ad3d4f49682a70f072"
server_key_version: "encryptionkey@0"
I20260924 03:20:10.290074 1611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "1a852df683db4628b3a2491d26680b98"
format_stamp: "Formatted at 2026-09-24 03:20:10 on dist-test-slave-vd0p"
server_key: "027267bfef244ec84aa8cde3ae1adbcc"
server_key_iv: "03147fc8aa34d6ad3d4f49682a70f072"
server_key_version: "encryptionkey@0"
I20260924 03:20:10.295156 1611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 03:20:10.298619 1625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:10.299906 1611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:20:10.300083 1611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "1a852df683db4628b3a2491d26680b98"
format_stamp: "Formatted at 2026-09-24 03:20:10 on dist-test-slave-vd0p"
server_key: "027267bfef244ec84aa8cde3ae1adbcc"
server_key_iv: "03147fc8aa34d6ad3d4f49682a70f072"
server_key_version: "encryptionkey@0"
I20260924 03:20:10.300305 1611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:10.317147 1611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:10.318030 1611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:10.318349 1611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:10.319278 1611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:10.320772 1611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:10.320904 1611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:10.321026 1611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:10.321111 1611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:10.354662 1611 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:46267
I20260924 03:20:10.354709 1737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:46267 every 8 connection(s)
I20260924 03:20:10.356001 1611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:20:10.360800 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 1611
I20260924 03:20:10.360999 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:20:10.361608 15246 external_mini_cluster.cc:1428] Setting key 28584d95c50e64e26082e7c98430f1e6
I20260924 03:20:10.364480 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46413
--builtin_ntp_servers=127.14.227.148:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:20:10.370339 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:10.358726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:45995 (local address 127.14.227.190:46413)
0924 03:20:10.359185 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:10.359189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:10.360326 (+ 1137us) server_negotiation.cc:407] Connection header received
0924 03:20:10.362692 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:10.362697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:10.362787 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:10.362911 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:10.364124 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:10.364928 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:10.367448 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:10.367917 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:10.368014 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:10.369569 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:10.369574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:10.369577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:10.369594 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:10.369640 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:10.369651 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:10.369717 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:10.370019 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 03:20:10.370093 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":230,"threads_started":1}
I20260924 03:20:10.371757 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:10.359315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46413 (local address 127.14.227.129:45995)
0924 03:20:10.360164 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:10.360203 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:10.362409 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:10.363141 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:10.363148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:10.363236 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:10.363950 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:10.363974 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:10.365102 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:10.365110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:10.367315 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:10.367332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:10.368135 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:10.368143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:10.368468 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:10.369201 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:10.369206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:10.369284 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:10.369381 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:10.369751 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:10.369765 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:10.369776 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:20:10.369978 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 03:20:10.370393 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":255,"threads_started":1}
I20260924 03:20:10.373462 1738 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46413
I20260924 03:20:10.373853 1738 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:10.374686 1738 heartbeater.cc:511] Master 127.14.227.190:46413 requested a full tablet report, sending...
I20260924 03:20:10.376929 1552 ts_manager.cc:194] Registered new tserver with Master: 1a852df683db4628b3a2491d26680b98 (127.14.227.129:46267)
I20260924 03:20:10.378558 1552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.129:45995
W20260924 03:20:10.693389 1742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:10.693717 1742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:10.693879 1742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:10.700976 1742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:10.701099 1742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:10.701270 1742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:20:10.710619 1742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:10.711974 1742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:10.713687 1742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:10.726337 1747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:10.726397 1750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:10.726368 1748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:10.727073 1742 server_base.cc:1034] running on GCE node
I20260924 03:20:10.727676 1742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:10.728473 1742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:10.729694 1742 hybrid_clock.cc:648] HybridClock initialized: now 1790220010729625 us; error 51 us; skew 500 ppm
I20260924 03:20:10.730028 1742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:10.732108 1742 webserver.cc:460] Webserver started at http://127.14.227.130:42055/ using document root <none> and password file <none>
I20260924 03:20:10.732700 1742 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:10.732776 1742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:10.733039 1742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:10.735853 1742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "2ebec4230a9d489d82de16913c6113f8"
format_stamp: "Formatted at 2026-09-24 03:20:10 on dist-test-slave-vd0p"
server_key: "b6cbc15bd1d922017883e14316771d5c"
server_key_iv: "9145382118a4c46a3cb1fee50d19df1e"
server_key_version: "encryptionkey@0"
I20260924 03:20:10.736558 1742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "2ebec4230a9d489d82de16913c6113f8"
format_stamp: "Formatted at 2026-09-24 03:20:10 on dist-test-slave-vd0p"
server_key: "b6cbc15bd1d922017883e14316771d5c"
server_key_iv: "9145382118a4c46a3cb1fee50d19df1e"
server_key_version: "encryptionkey@0"
I20260924 03:20:10.741382 1742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:20:10.744606 1756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:10.745814 1742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 03:20:10.745932 1742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "2ebec4230a9d489d82de16913c6113f8"
format_stamp: "Formatted at 2026-09-24 03:20:10 on dist-test-slave-vd0p"
server_key: "b6cbc15bd1d922017883e14316771d5c"
server_key_iv: "9145382118a4c46a3cb1fee50d19df1e"
server_key_version: "encryptionkey@0"
I20260924 03:20:10.746145 1742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:10.765431 1742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:10.766184 1742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:10.766417 1742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:10.767319 1742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:10.768568 1742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:10.768646 1742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:10.768714 1742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:10.768766 1742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:10.800650 1742 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:39453
I20260924 03:20:10.800711 1868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:39453 every 8 connection(s)
I20260924 03:20:10.802150 1742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:20:10.811239 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 1742
I20260924 03:20:10.811417 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:20:10.811970 15246 external_mini_cluster.cc:1428] Setting key 9ce1eb71fbf3082b52a9cb693c5d3776
I20260924 03:20:10.815047 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46413
--builtin_ntp_servers=127.14.227.148:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:20:10.815635 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:10.804730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:60201 (local address 127.14.227.190:46413)
0924 03:20:10.804902 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:10.804907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:10.806548 (+ 1641us) server_negotiation.cc:407] Connection header received
0924 03:20:10.808321 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:10.808327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:10.808418 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:10.808529 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:10.809563 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:10.810364 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:10.812536 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:10.813239 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:10.813393 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:10.814887 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:10.814893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:10.814896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:10.814912 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:10.814940 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:10.814951 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:10.815044 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:10.815335 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 03:20:10.815419 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":60}
I20260924 03:20:10.817030 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:10.805383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46413 (local address 127.14.227.130:60201)
0924 03:20:10.806388 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:10.806422 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:10.808068 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:10.808761 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:10.808769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:10.808851 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:10.809407 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:10.809422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:10.810515 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:10.810521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:10.812358 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:10.812374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:10.813512 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:10.813520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:10.813823 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:10.814586 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:10.814589 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:10.814625 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:10.814700 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:10.815043 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:10.815056 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:10.815078 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:20:10.815832 (+ 754us) client_negotiation.cc:240] Negotiation successful
0924 03:20:10.816145 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":300,"threads_started":1}
I20260924 03:20:10.818382 1869 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46413
I20260924 03:20:10.818677 1869 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:10.819388 1869 heartbeater.cc:511] Master 127.14.227.190:46413 requested a full tablet report, sending...
I20260924 03:20:10.820672 1552 ts_manager.cc:194] Registered new tserver with Master: 2ebec4230a9d489d82de16913c6113f8 (127.14.227.130:39453)
I20260924 03:20:10.821483 1552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.130:60201
W20260924 03:20:11.133060 1872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:11.133358 1872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:11.133522 1872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:11.140707 1872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:11.140815 1872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:11.140986 1872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:20:11.149770 1872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:11.150972 1872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:11.152706 1872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:11.165684 1877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:11.165729 1878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:11.165759 1880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:11.166617 1872 server_base.cc:1034] running on GCE node
I20260924 03:20:11.167265 1872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:11.168166 1872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:11.169389 1872 hybrid_clock.cc:648] HybridClock initialized: now 1790220011169331 us; error 24 us; skew 500 ppm
I20260924 03:20:11.169696 1872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:11.172077 1872 webserver.cc:460] Webserver started at http://127.14.227.131:41709/ using document root <none> and password file <none>
I20260924 03:20:11.172760 1872 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:11.172874 1872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:11.173179 1872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:11.175940 1872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "646db4040618446b8d7918e0401a925a"
format_stamp: "Formatted at 2026-09-24 03:20:11 on dist-test-slave-vd0p"
server_key: "071591f9f5593728f8d404e41f0b6c7e"
server_key_iv: "51df1886295cd01aa5bcb3c3981d54d5"
server_key_version: "encryptionkey@0"
I20260924 03:20:11.176712 1872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "646db4040618446b8d7918e0401a925a"
format_stamp: "Formatted at 2026-09-24 03:20:11 on dist-test-slave-vd0p"
server_key: "071591f9f5593728f8d404e41f0b6c7e"
server_key_iv: "51df1886295cd01aa5bcb3c3981d54d5"
server_key_version: "encryptionkey@0"
I20260924 03:20:11.181732 1872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260924 03:20:11.184911 1886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:11.186180 1872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:20:11.186338 1872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "646db4040618446b8d7918e0401a925a"
format_stamp: "Formatted at 2026-09-24 03:20:11 on dist-test-slave-vd0p"
server_key: "071591f9f5593728f8d404e41f0b6c7e"
server_key_iv: "51df1886295cd01aa5bcb3c3981d54d5"
server_key_version: "encryptionkey@0"
I20260924 03:20:11.186535 1872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:11.199602 1872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:11.200337 1872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:11.200629 1872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:11.201530 1872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:11.202877 1872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:11.202961 1872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:11.203109 1872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:11.203188 1872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:11.235662 1872 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:32775
I20260924 03:20:11.235754 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:32775 every 8 connection(s)
I20260924 03:20:11.236987 1872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:20:11.239926 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 1872
I20260924 03:20:11.240104 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:20:11.240527 15246 external_mini_cluster.cc:1428] Setting key 2d3fbbd3df731d02d2fe2ece35214654
I20260924 03:20:11.249661 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.239715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:37939 (local address 127.14.227.190:46413)
0924 03:20:11.239936 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.239942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.241532 (+ 1590us) server_negotiation.cc:407] Connection header received
0924 03:20:11.243758 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.243762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.243826 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.243924 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:11.244994 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.245774 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.247278 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.247651 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.247742 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.248939 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:11.248944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:11.248946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:11.248962 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:11.248991 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:11.249003 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:11.249066 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.249370 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.249441 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 03:20:11.250698 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.240345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:46413 (local address 127.14.227.131:37939)
0924 03:20:11.241380 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.241431 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:11.243516 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:11.244231 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:11.244239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:11.244325 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:11.244852 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.244868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.245901 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.245905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.247160 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.247171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.247819 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.247823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.248012 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.248622 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:11.248625 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:11.248659 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:11.248768 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:11.249126 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:11.249135 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:11.249143 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 03:20:11.249340 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 03:20:11.249604 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":316,"threads_started":1}
I20260924 03:20:11.251935 1999 heartbeater.cc:347] Connected to a master server at 127.14.227.190:46413
I20260924 03:20:11.252223 1999 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:11.252910 1999 heartbeater.cc:511] Master 127.14.227.190:46413 requested a full tablet report, sending...
I20260924 03:20:11.254294 1552 ts_manager.cc:194] Registered new tserver with Master: 646db4040618446b8d7918e0401a925a (127.14.227.131:32775)
I20260924 03:20:11.255262 1552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.131:37939
I20260924 03:20:11.255944 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:20:11.266105 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.260460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.14.227.190:46413)
0924 03:20:11.260678 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.260682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.261055 (+ 373us) server_negotiation.cc:407] Connection header received
0924 03:20:11.261217 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.261220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.261296 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.261373 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:11.261825 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.262575 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.263878 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.264270 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.264356 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.265407 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:11.265412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:11.265415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:11.265430 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:11.265457 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:11.265468 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:11.265567 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.265771 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.265836 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:20:11.269397 1552 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:20:11.277508 1552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:20:11.311934 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.290927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.14.227.129:46267)
0924 03:20:11.303727 (+ 12800us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.303739 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.303778 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:20:11.303892 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.303904 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.304311 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.304534 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:11.305339 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.306836 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.309379 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.310705 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.310858 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.311036 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.311445 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.311599 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12640,"thread_start_us":12497,"threads_started":1}
I20260924 03:20:11.311908 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.290320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.0.0.1:46652)
0924 03:20:11.302764 (+ 12444us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.302796 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:11.302949 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:11.304574 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:11.304583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:11.304621 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:11.305108 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.305126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.306977 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.306983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.309148 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.309164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.311122 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.311126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.311249 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.311279 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 03:20:11.311404 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 03:20:11.311597 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12253,"thread_start_us":10033,"threads_started":1}
I20260924 03:20:11.313201 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.290087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:32775 (local address 127.0.0.1:56918)
0924 03:20:11.300843 (+ 10756us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.300910 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:11.301127 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:11.306500 (+ 5373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:11.306505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:11.306530 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:11.306968 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.306978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.308681 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.308686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.310669 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.310683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.312506 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.312511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.312652 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.312679 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 03:20:11.312760 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:20:11.312875 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10459,"thread_start_us":13800,"threads_started":1}
I20260924 03:20:11.313498 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.290685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.14.227.131:32775)
0924 03:20:11.305854 (+ 15169us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.305865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.305900 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:20:11.306028 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.306037 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.306258 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.306498 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:11.307194 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.308510 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.310842 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.312275 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.312519 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.312767 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.312841 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.313028 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14973,"thread_start_us":14766,"threads_started":1}
I20260924 03:20:11.314471 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.290704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:39453 (local address 127.0.0.1:35152)
0924 03:20:11.304203 (+ 13499us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.304230 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:11.304359 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:11.306007 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:11.306012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:11.306037 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:11.306356 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.306369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.308155 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.308161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.310285 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.310298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.313487 (+ 3189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.313500 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.313725 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.313772 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:20:11.313880 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 03:20:11.314074 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13369,"mutex_wait_us":1,"thread_start_us":12368,"threads_started":1}
I20260924 03:20:11.314503 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.291546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.14.227.130:39453)
0924 03:20:11.305318 (+ 13772us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.305341 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.305378 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:20:11.305494 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.305505 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.305763 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.306015 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:11.306547 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.308000 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.310551 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.312655 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.312860 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.313112 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.313982 (+ 870us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.314158 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13528,"thread_start_us":13340,"threads_started":1}
I20260924 03:20:11.316176 1673 tablet_service.cc:1467] Processing CreateTablet for tablet c206dd27641d4f8ba923eba8e9312c8f (DEFAULT_TABLE table=test-table [id=16d0613759e044dc9f82cc0dd60b6094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:11.317577 1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c206dd27641d4f8ba923eba8e9312c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:11.317171 1934 tablet_service.cc:1467] Processing CreateTablet for tablet c206dd27641d4f8ba923eba8e9312c8f (DEFAULT_TABLE table=test-table [id=16d0613759e044dc9f82cc0dd60b6094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:11.317883 1804 tablet_service.cc:1467] Processing CreateTablet for tablet c206dd27641d4f8ba923eba8e9312c8f (DEFAULT_TABLE table=test-table [id=16d0613759e044dc9f82cc0dd60b6094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:11.318708 1934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c206dd27641d4f8ba923eba8e9312c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:11.319195 1804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c206dd27641d4f8ba923eba8e9312c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:11.330731 2017 tablet_bootstrap.cc:492] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Bootstrap starting.
I20260924 03:20:11.334818 2018 tablet_bootstrap.cc:492] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Bootstrap starting.
I20260924 03:20:11.335479 2017 tablet_bootstrap.cc:654] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:11.336999 2017 log.cc:824] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:11.338685 2019 tablet_bootstrap.cc:492] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Bootstrap starting.
I20260924 03:20:11.339141 2018 tablet_bootstrap.cc:654] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:11.340535 2018 log.cc:824] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:11.342675 2019 tablet_bootstrap.cc:654] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:11.343755 2017 tablet_bootstrap.cc:492] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: No bootstrap required, opened a new log
I20260924 03:20:11.343873 2018 tablet_bootstrap.cc:492] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: No bootstrap required, opened a new log
I20260924 03:20:11.344094 2017 ts_tablet_manager.cc:1397] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260924 03:20:11.344184 2018 ts_tablet_manager.cc:1397] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 03:20:11.344187 2019 log.cc:824] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:11.347357 2019 tablet_bootstrap.cc:492] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: No bootstrap required, opened a new log
I20260924 03:20:11.347641 2019 ts_tablet_manager.cc:1397] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 03:20:11.351557 2017 raft_consensus.cc:348] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.351919 2017 raft_consensus.cc:374] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:11.352041 2017 raft_consensus.cc:729] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646db4040618446b8d7918e0401a925a, State: Initialized, Role: FOLLOWER
I20260924 03:20:11.352651 2017 consensus_queue.cc:260] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.353739 1999 heartbeater.cc:503] Master 127.14.227.190:46413 was elected leader, sending a full tablet report...
I20260924 03:20:11.354399 2017 ts_tablet_manager.cc:1428] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Time spent starting tablet: real 0.010s user 0.008s sys 0.005s
I20260924 03:20:11.356345 2018 raft_consensus.cc:348] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.356841 2018 raft_consensus.cc:374] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:11.356966 2018 raft_consensus.cc:729] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a852df683db4628b3a2491d26680b98, State: Initialized, Role: FOLLOWER
I20260924 03:20:11.357708 2018 consensus_queue.cc:260] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.359972 2019 raft_consensus.cc:348] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.360368 2019 raft_consensus.cc:374] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:11.360463 2019 raft_consensus.cc:729] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebec4230a9d489d82de16913c6113f8, State: Initialized, Role: FOLLOWER
I20260924 03:20:11.361013 2019 consensus_queue.cc:260] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.364516 1738 heartbeater.cc:503] Master 127.14.227.190:46413 was elected leader, sending a full tablet report...
I20260924 03:20:11.365227 2018 ts_tablet_manager.cc:1428] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
I20260924 03:20:11.366658 1869 heartbeater.cc:503] Master 127.14.227.190:46413 was elected leader, sending a full tablet report...
I20260924 03:20:11.367152 2019 ts_tablet_manager.cc:1428] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
W20260924 03:20:11.489092 2000 tablet.cc:2369] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:20:11.554317 1870 tablet.cc:2369] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:11.574600 2023 raft_consensus.cc:482] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:20:11.574878 2023 raft_consensus.cc:504] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.576589 2023 leader_election.cc:288] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a852df683db4628b3a2491d26680b98 (127.14.227.129:46267), 2ebec4230a9d489d82de16913c6113f8 (127.14.227.130:39453)
I20260924 03:20:11.583099 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.577049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:36209 (local address 127.14.227.129:46267)
0924 03:20:11.577294 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.577303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.577330 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:20:11.577525 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.577529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.577600 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.577708 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:11.578379 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.579274 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.581256 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.582219 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.582343 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.582391 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.582708 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.582829 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:20:11.583148 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.576833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.14.227.131:36209)
0924 03:20:11.577148 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.577179 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:11.577316 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:11.577791 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:11.577797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:11.577830 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:11.578192 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.578204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.579405 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.579411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.581084 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.581096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.582364 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.582370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.582544 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.582577 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 03:20:11.582665 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 03:20:11.582801 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"spinlock_wait_cycles":39296}
I20260924 03:20:11.583499 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.577207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:33821 (local address 127.14.227.130:39453)
0924 03:20:11.577406 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.577412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.577973 (+ 561us) server_negotiation.cc:407] Connection header received
0924 03:20:11.578176 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.578181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.578260 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.578361 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:11.578914 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.579747 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.581633 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.582565 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.582725 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.582787 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.583133 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.583271 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:20:11.583794 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.577066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:39453 (local address 127.14.227.131:33821)
0924 03:20:11.577826 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.577839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:11.578039 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:11.578384 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:11.578387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:11.578410 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:11.578747 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.578758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.579894 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.579898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.581449 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:11.581462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.582754 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.582760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:11.582954 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.582989 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:20:11.583082 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:20:11.583216 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":446,"threads_started":1}
I20260924 03:20:11.584476 1693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c206dd27641d4f8ba923eba8e9312c8f" candidate_uuid: "646db4040618446b8d7918e0401a925a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a852df683db4628b3a2491d26680b98" is_pre_election: true
I20260924 03:20:11.584663 1824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c206dd27641d4f8ba923eba8e9312c8f" candidate_uuid: "646db4040618446b8d7918e0401a925a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebec4230a9d489d82de16913c6113f8" is_pre_election: true
I20260924 03:20:11.585023 1693 raft_consensus.cc:2454] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 646db4040618446b8d7918e0401a925a in term 0.
I20260924 03:20:11.585191 1824 raft_consensus.cc:2454] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 646db4040618446b8d7918e0401a925a in term 0.
I20260924 03:20:11.585757 1888 leader_election.cc:302] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a852df683db4628b3a2491d26680b98, 2ebec4230a9d489d82de16913c6113f8, 646db4040618446b8d7918e0401a925a; no voters:
I20260924 03:20:11.586210 2023 raft_consensus.cc:2781] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:20:11.586330 2023 raft_consensus.cc:482] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:20:11.586441 2023 raft_consensus.cc:3037] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:11.588753 2023 raft_consensus.cc:504] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.589402 2023 leader_election.cc:288] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [CANDIDATE]: Term 1 election: Requested vote from peers 1a852df683db4628b3a2491d26680b98 (127.14.227.129:46267), 2ebec4230a9d489d82de16913c6113f8 (127.14.227.130:39453)
I20260924 03:20:11.589826 1693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c206dd27641d4f8ba923eba8e9312c8f" candidate_uuid: "646db4040618446b8d7918e0401a925a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a852df683db4628b3a2491d26680b98"
I20260924 03:20:11.589855 1824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c206dd27641d4f8ba923eba8e9312c8f" candidate_uuid: "646db4040618446b8d7918e0401a925a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ebec4230a9d489d82de16913c6113f8"
I20260924 03:20:11.590090 1824 raft_consensus.cc:3037] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:11.590083 1693 raft_consensus.cc:3037] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:11.592339 1693 raft_consensus.cc:2454] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 646db4040618446b8d7918e0401a925a in term 1.
I20260924 03:20:11.592362 1824 raft_consensus.cc:2454] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 646db4040618446b8d7918e0401a925a in term 1.
I20260924 03:20:11.592695 1888 leader_election.cc:302] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a852df683db4628b3a2491d26680b98, 646db4040618446b8d7918e0401a925a; no voters:
I20260924 03:20:11.592965 2023 raft_consensus.cc:2781] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:11.593369 2023 raft_consensus.cc:686] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 1 LEADER]: Becoming Leader. State: Replica: 646db4040618446b8d7918e0401a925a, State: Running, Role: LEADER
I20260924 03:20:11.593822 2023 consensus_queue.cc:237] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } }
I20260924 03:20:11.598078 1552 catalog_manager.cc:5495] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a reported cstate change: term changed from 0 to 1, leader changed from <none> to 646db4040618446b8d7918e0401a925a (127.14.227.131). New cstate: current_term: 1 leader_uuid: "646db4040618446b8d7918e0401a925a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "646db4040618446b8d7918e0401a925a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 32775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 } health_report { overall_health: UNKNOWN } } }
W20260924 03:20:11.608494 1739 tablet.cc:2369] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:11.674803 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.668081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.14.227.131:32775)
0924 03:20:11.668337 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.668344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.668406 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:20:11.668697 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.668710 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.668811 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.668993 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:11.669454 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.670320 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.672016 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.672574 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.672745 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.673488 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:11.674049 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:11.674285 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.674483 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.674547 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 03:20:11.684263 1824 raft_consensus.cc:1261] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 1 FOLLOWER]: Refusing update from remote peer 646db4040618446b8d7918e0401a925a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:11.684235 1693 raft_consensus.cc:1261] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 1 FOLLOWER]: Refusing update from remote peer 646db4040618446b8d7918e0401a925a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:11.685359 2027 consensus_queue.cc:1035] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ebec4230a9d489d82de16913c6113f8" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:11.685725 2023 consensus_queue.cc:1035] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a852df683db4628b3a2491d26680b98" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:11.698499 2032 mvcc.cc:205] Tried to move back new op lower bound from 7332741167847247872 to 7332741167491846144. Current Snapshot: MvccSnapshot[applied={T|T < 7332741167847247872}]
I20260924 03:20:11.706264 2031 mvcc.cc:205] Tried to move back new op lower bound from 7332741167847247872 to 7332741167491846144. Current Snapshot: MvccSnapshot[applied={T|T < 7332741167847247872 or (T in {7332741167847247872})}]
I20260924 03:20:11.711380 1551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:11.711614 1551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:11.715730 1551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac590be141e643cf9866e489cd1b0f4f: Sending DeleteTablet for 3 replicas of tablet c206dd27641d4f8ba923eba8e9312c8f
I20260924 03:20:11.717121 1804 tablet_service.cc:1514] Processing DeleteTablet for tablet c206dd27641d4f8ba923eba8e9312c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:11 UTC) from {username='slave'} at 127.0.0.1:35152
I20260924 03:20:11.717121 1673 tablet_service.cc:1514] Processing DeleteTablet for tablet c206dd27641d4f8ba923eba8e9312c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:11 UTC) from {username='slave'} at 127.0.0.1:46652
I20260924 03:20:11.717356 1934 tablet_service.cc:1514] Processing DeleteTablet for tablet c206dd27641d4f8ba923eba8e9312c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:11 UTC) from {username='slave'} at 127.0.0.1:56918
I20260924 03:20:11.719527 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 1611
I20260924 03:20:11.732638 2041 tablet_replica.cc:330] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: stopping tablet replica
I20260924 03:20:11.733218 2041 raft_consensus.cc:2229] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:20:11.734212 2041 raft_consensus.cc:2258] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:11.735766 2040 tablet_replica.cc:330] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: stopping tablet replica
I20260924 03:20:11.736222 2040 raft_consensus.cc:2229] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:11.736814 2040 raft_consensus.cc:2258] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:11.739313 2040 ts_tablet_manager.cc:1905] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:11.741117 2041 ts_tablet_manager.cc:1905] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:11.746574 2040 ts_tablet_manager.cc:1918] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:11.747013 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.721230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.14.227.129:46267)
0924 03:20:11.721462 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.721471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.735075 (+ 13604us) server_negotiation.cc:407] Connection header received
0924 03:20:11.735281 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.735292 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.735456 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.735577 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:11.736274 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.737520 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.742865 (+ 5345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.743573 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.743678 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.745265 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:11.745276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:11.745293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:11.746076 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:11.746131 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:11.746160 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:11.746247 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.746503 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.746628 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 03:20:11.746742 2040 log.cc:1196] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/c206dd27641d4f8ba923eba8e9312c8f
I20260924 03:20:11.747828 1733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:11.748030 2039 tablet_replica.cc:330] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: stopping tablet replica
I20260924 03:20:11.748467 2039 raft_consensus.cc:2229] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:11.749027 2039 raft_consensus.cc:2258] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:11.751338 2039 ts_tablet_manager.cc:1905] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:11.753829 2041 ts_tablet_manager.cc:1918] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:11.754041 2041 log.cc:1196] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/c206dd27641d4f8ba923eba8e9312c8f
I20260924 03:20:11.754603 2041 ts_tablet_manager.cc:1939] T c206dd27641d4f8ba923eba8e9312c8f P 646db4040618446b8d7918e0401a925a: Deleting consensus metadata
I20260924 03:20:11.756510 1536 catalog_manager.cc:4841] TS 646db4040618446b8d7918e0401a925a (127.14.227.131:32775): tablet c206dd27641d4f8ba923eba8e9312c8f (table test-table [id=16d0613759e044dc9f82cc0dd60b6094]) successfully deleted
I20260924 03:20:11.759259 2039 ts_tablet_manager.cc:1918] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:11.759419 2039 log.cc:1196] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/c206dd27641d4f8ba923eba8e9312c8f
I20260924 03:20:11.759922 2039 ts_tablet_manager.cc:1939] T c206dd27641d4f8ba923eba8e9312c8f P 2ebec4230a9d489d82de16913c6113f8: Deleting consensus metadata
I20260924 03:20:11.761754 1539 catalog_manager.cc:4841] TS 2ebec4230a9d489d82de16913c6113f8 (127.14.227.130:39453): tablet c206dd27641d4f8ba923eba8e9312c8f (table test-table [id=16d0613759e044dc9f82cc0dd60b6094]) successfully deleted
I20260924 03:20:11.859112 2040 ts_tablet_manager.cc:1939] T c206dd27641d4f8ba923eba8e9312c8f P 1a852df683db4628b3a2491d26680b98: Deleting consensus metadata
I20260924 03:20:11.859334 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 1611
I20260924 03:20:11.876655 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 1742
W20260924 03:20:11.876791 1537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.227.129:46267 (error 108)
I20260924 03:20:11.879299 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.878413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.0.0.1:35964)
0924 03:20:11.878895 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.879030 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":233,"threads_started":1}
W20260924 03:20:11.879588 1537 catalog_manager.cc:4584] TS 1a852df683db4628b3a2491d26680b98 (127.14.227.129:46267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c206dd27641d4f8ba923eba8e9312c8f: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46267: connect: Connection refused (error 111)
I20260924 03:20:11.885238 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.877506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.14.227.130:39453)
0924 03:20:11.877721 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.877729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.877755 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:20:11.878001 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.878007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.878114 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.878227 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:11.878816 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.879840 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.881941 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.882510 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.882608 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:11.883863 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:11.883871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:11.883886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:11.884465 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:11.884510 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:11.884528 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:11.884620 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:11.884802 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 03:20:11.884895 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:20:11.885856 1864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:11.933254 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.932671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.0.0.1:35976)
0924 03:20:11.932944 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.933063 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 03:20:11.934562 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.934096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.0.0.1:35982)
0924 03:20:11.934280 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.934373 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 03:20:11.977025 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 1742
I20260924 03:20:11.994349 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 1872
I20260924 03:20:11.995393 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.994878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.0.0.1:35986)
0924 03:20:11.995108 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.995216 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 03:20:11.996503 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.996119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.0.0.1:35994)
0924 03:20:11.996265 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:11.996345 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 03:20:12.001778 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:11.995243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.14.227.131:32775)
0924 03:20:11.995422 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:11.995428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:11.995490 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:20:11.995779 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:11.995785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:11.995881 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:11.995995 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:11.996469 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.997477 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.998819 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:11.999258 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:11.999391 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:12.000438 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:12.000447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:12.000463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:12.001037 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:12.001083 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:12.001101 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:12.001219 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:12.001357 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 03:20:12.001453 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:20:12.002485 1994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:12.077432 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:12.076965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.0.0.1:36008)
0924 03:20:12.077178 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:12.077289 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 03:20:12.078569 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:12.078132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:46267 (local address 127.0.0.1:36024)
0924 03:20:12.078365 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:12.078435 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 03:20:12.120486 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 1872
I20260924 03:20:12.139546 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 1521
I20260924 03:20:12.140612 1582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:12.249874 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 1521
2026-09-24T03:20:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2938 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:12 dist-test-slave-vd0p krb5kdc[2051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:12 dist-test-slave-vd0p krb5kdc[2051](info): set up 2 sockets
Sep 24 03:20:12 dist-test-slave-vd0p krb5kdc[2051](info): commencing operation
krb5kdc: starting...
W20260924 03:20:13.402406 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:13 dist-test-slave-vd0p krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:13.637197 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:45385
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:45385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:13.957827 2064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:13.958191 2064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:13.958362 2064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:13.965338 2064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:13.965533 2064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:13.965634 2064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:13.965739 2064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:13.965828 2064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:13.975106 2064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:45385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:45385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:13.976385 2064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:13.977669 2064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:13.989387 2069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:13.989703 2070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:13.989698 2072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:13.990569 2064 server_base.cc:1034] running on GCE node
I20260924 03:20:13.991222 2064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:13.992483 2064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:13.993706 2064 hybrid_clock.cc:648] HybridClock initialized: now 1790220013993667 us; error 50 us; skew 500 ppm
I20260924 03:20:13.994077 2064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:14 dist-test-slave-vd0p krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220014, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:14.004768 2064 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:14.006379 2064 webserver.cc:460] Webserver started at http://127.14.227.190:36449/ using document root <none> and password file <none>
I20260924 03:20:14.007047 2064 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:14.007215 2064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:14.007509 2064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:14.010380 2064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "9745222d8df74c85a81bf2aaa1c02513"
format_stamp: "Formatted at 2026-09-24 03:20:14 on dist-test-slave-vd0p"
server_key: "4696bafccef77926103435ae4a3bcb74"
server_key_iv: "e169f0688d4ffe0165623d3ef06c4f5a"
server_key_version: "encryptionkey@0"
I20260924 03:20:14.011124 2064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "9745222d8df74c85a81bf2aaa1c02513"
format_stamp: "Formatted at 2026-09-24 03:20:14 on dist-test-slave-vd0p"
server_key: "4696bafccef77926103435ae4a3bcb74"
server_key_iv: "e169f0688d4ffe0165623d3ef06c4f5a"
server_key_version: "encryptionkey@0"
I20260924 03:20:14.016142 2064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:20:14.019547 2079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:14.020774 2064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:20:14.020957 2064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "9745222d8df74c85a81bf2aaa1c02513"
format_stamp: "Formatted at 2026-09-24 03:20:14 on dist-test-slave-vd0p"
server_key: "4696bafccef77926103435ae4a3bcb74"
server_key_iv: "e169f0688d4ffe0165623d3ef06c4f5a"
server_key_version: "encryptionkey@0"
I20260924 03:20:14.021162 2064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:14.039822 2064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:14.044234 2064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:14.044523 2064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:14.064363 2064 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:45385
I20260924 03:20:14.064380 2130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:45385 every 8 connection(s)
I20260924 03:20:14.065711 2064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:14.069311 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:14.070163 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 2064
I20260924 03:20:14.070355 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:14.070914 15246 external_mini_cluster.cc:1428] Setting key 6cbc90d6e4dd530c3a1e1f846011e15e
I20260924 03:20:14.083508 2131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513: Bootstrap starting.
Sep 24 03:20:14 dist-test-slave-vd0p krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:14.087123 2131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:14.088493 2131 log.cc:824] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:14.091684 2131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513: No bootstrap required, opened a new log
I20260924 03:20:14.094877 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:14.072750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.14.227.190:45385)
0924 03:20:14.073487 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:14.073498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:14.073537 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:20:14.075647 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:14.075669 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:14.076104 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:14.076445 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:14.078261 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:14.079630 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:14.081427 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:14.082156 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:14.082286 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:14.086832 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:14.086843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:14.086862 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:14.086912 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:14.090573 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:14.091201 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:14.091210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:14.091217 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:14.091404 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:14.091757 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:14.091762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:14.091765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:14.092456 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:14.092738 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:14.093090 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 03:20:14.093514 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":335,"threads_started":1}
I20260924 03:20:14.101177 2131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9745222d8df74c85a81bf2aaa1c02513" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45385 } }
I20260924 03:20:14.101511 2131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:14.101588 2131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9745222d8df74c85a81bf2aaa1c02513, State: Initialized, Role: FOLLOWER
I20260924 03:20:14.102108 2131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9745222d8df74c85a81bf2aaa1c02513" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45385 } }
I20260924 03:20:14.102313 2131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:14.102407 2131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:14.102536 2131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:14.104758 2131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9745222d8df74c85a81bf2aaa1c02513" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45385 } }
I20260924 03:20:14.105180 2131 leader_election.cc:302] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9745222d8df74c85a81bf2aaa1c02513; no voters:
I20260924 03:20:14.105674 2131 leader_election.cc:288] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:14.105824 2136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:14.106521 2136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [term 1 LEADER]: Becoming Leader. State: Replica: 9745222d8df74c85a81bf2aaa1c02513, State: Running, Role: LEADER
I20260924 03:20:14.107002 2136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9745222d8df74c85a81bf2aaa1c02513" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45385 } }
I20260924 03:20:14.107710 2131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:14.109863 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9745222d8df74c85a81bf2aaa1c02513. Latest consensus state: current_term: 1 leader_uuid: "9745222d8df74c85a81bf2aaa1c02513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9745222d8df74c85a81bf2aaa1c02513" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45385 } } }
I20260924 03:20:14.110178 2137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9745222d8df74c85a81bf2aaa1c02513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9745222d8df74c85a81bf2aaa1c02513" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45385 } } }
I20260924 03:20:14.110368 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:14.110469 2137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:14.114162 2144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:14.118438 2144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:14.127133 2144 catalog_manager.cc:1347] Generated new cluster ID: 16bc64c79b334e0784fa33c3c802acf4
I20260924 03:20:14.127241 2144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:14.142061 2144 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:14.143771 2144 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:14.154455 2144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513: Generated new TSK 0
I20260924 03:20:14.155433 2144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.227.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
Entry for principal kudu/127.14.227.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab.
I20260924 03:20:14.215698 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45385
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:14.539079 2157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:14.539386 2157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:14.539556 2157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:14.546476 2157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:14.546612 2157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:14.546799 2157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:20:14.555641 2157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.129.keytab
--principal=kudu/127.14.227.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:14.556869 2157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:14.558143 2157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:14.570739 2162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:14.570706 2163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:14.570904 2165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:14.571918 2157 server_base.cc:1034] running on GCE node
I20260924 03:20:14.572448 2157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:14.573307 2157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:14.574530 2157 hybrid_clock.cc:648] HybridClock initialized: now 1790220014574478 us; error 69 us; skew 500 ppm
I20260924 03:20:14.574872 2157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:14 dist-test-slave-vd0p krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220014, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:14.577929 2157 init.cc:377] Logged in from keytab as kudu/127.14.227.129@KRBTEST.COM (short username kudu)
I20260924 03:20:14.579294 2157 webserver.cc:460] Webserver started at http://127.14.227.129:36281/ using document root <none> and password file <none>
I20260924 03:20:14.579921 2157 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:14.580061 2157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:14.580366 2157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:14.583133 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "7bfbd5c683d04c4b808ed904e4aef4c8"
format_stamp: "Formatted at 2026-09-24 03:20:14 on dist-test-slave-vd0p"
server_key: "aa35e0b484f56768c5d737a2ace20629"
server_key_iv: "38fe7c8b9d8637572810f88802d41187"
server_key_version: "encryptionkey@0"
I20260924 03:20:14.583853 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "7bfbd5c683d04c4b808ed904e4aef4c8"
format_stamp: "Formatted at 2026-09-24 03:20:14 on dist-test-slave-vd0p"
server_key: "aa35e0b484f56768c5d737a2ace20629"
server_key_iv: "38fe7c8b9d8637572810f88802d41187"
server_key_version: "encryptionkey@0"
I20260924 03:20:14.588735 2157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 03:20:14.591962 2172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:14.593173 2157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:20:14.593359 2157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "7bfbd5c683d04c4b808ed904e4aef4c8"
format_stamp: "Formatted at 2026-09-24 03:20:14 on dist-test-slave-vd0p"
server_key: "aa35e0b484f56768c5d737a2ace20629"
server_key_iv: "38fe7c8b9d8637572810f88802d41187"
server_key_version: "encryptionkey@0"
I20260924 03:20:14.593565 2157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:14.617468 2157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:14.621570 2157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:14.621800 2157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:14.622722 2157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:14.623924 2157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:14.624004 2157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:14.624083 2157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:14.624132 2157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:14.656419 2157 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:45103
I20260924 03:20:14.656526 2284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:45103 every 8 connection(s)
I20260924 03:20:14.657784 2157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:20:14.660843 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 2157
I20260924 03:20:14.661037 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:20:14.661448 15246 external_mini_cluster.cc:1428] Setting key 801fca9eaedf4d42effd1d8886c82c03
Sep 24 03:20:14 dist-test-slave-vd0p krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220014, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.129@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:14.678208 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:14.660378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:35517 (local address 127.14.227.190:45385)
0924 03:20:14.660911 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:14.660919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:14.662021 (+ 1102us) server_negotiation.cc:407] Connection header received
0924 03:20:14.663774 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:14.663779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:14.663867 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:14.663986 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:14.665437 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:14.666239 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:14.667766 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:14.668168 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:14.668300 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:14.671965 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:14.671971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:14.671974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:14.672002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:14.674642 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:14.675416 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:14.675424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:14.675428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:14.675595 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:14.676219 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:14.676224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:14.676228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:14.676503 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:14.676835 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:14.677528 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 03:20:14.677755 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":238,"threads_started":1}
I20260924 03:20:14.679207 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:14.660952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45385 (local address 127.14.227.129:35517)
0924 03:20:14.661795 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:14.661846 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:14.663503 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:14.664209 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:14.664217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:14.664699 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:14.665268 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:14.665286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:14.666418 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:14.666422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:14.667597 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:14.667609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:14.668400 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:14.668409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:14.668643 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:14.669316 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:14.669337 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:14.671759 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:14.674796 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:14.674807 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:14.674822 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:14.675284 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:14.675890 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:14.675899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:14.675902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:14.676101 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:14.676811 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:14.676819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:14.677143 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 03:20:14.677423 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 03:20:14.678003 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":335,"threads_started":1}
I20260924 03:20:14.680814 2285 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45385
I20260924 03:20:14.681138 2285 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:14.681862 2285 heartbeater.cc:511] Master 127.14.227.190:45385 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.227.130@KRBTEST.COM; defaulting to no policy
I20260924 03:20:14.684150 2096 ts_manager.cc:194] Registered new tserver with Master: 7bfbd5c683d04c4b808ed904e4aef4c8 (127.14.227.129:45103)
I20260924 03:20:14.685782 2096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.129@KRBTEST.COM'} at 127.14.227.129:35517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
Entry for principal kudu/127.14.227.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab.
I20260924 03:20:14.714494 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45385
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:15.027694 2291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:15.027940 2291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:15.028036 2291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:15.035123 2291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:15.035221 2291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:15.035327 2291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:20:15.044340 2291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.130.keytab
--principal=kudu/127.14.227.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:15.045523 2291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:15.046825 2291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:15.059610 2299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:15.059705 2296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:15.059741 2297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:15.060748 2291 server_base.cc:1034] running on GCE node
I20260924 03:20:15.061305 2291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:15.062204 2291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:15.063397 2291 hybrid_clock.cc:648] HybridClock initialized: now 1790220015063351 us; error 39 us; skew 500 ppm
I20260924 03:20:15.063728 2291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:15 dist-test-slave-vd0p krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220015, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:15.066835 2291 init.cc:377] Logged in from keytab as kudu/127.14.227.130@KRBTEST.COM (short username kudu)
I20260924 03:20:15.068334 2291 webserver.cc:460] Webserver started at http://127.14.227.130:44433/ using document root <none> and password file <none>
I20260924 03:20:15.068961 2291 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:15.069080 2291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:15.069375 2291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:15.072032 2291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "ac155eacfc6346a9bdf11b055c20f0a7"
format_stamp: "Formatted at 2026-09-24 03:20:15 on dist-test-slave-vd0p"
server_key: "38d2ff7e6a8cbaf1bec0c05c1db0151c"
server_key_iv: "24a0a44dc9ae221bcbdcccf645666c28"
server_key_version: "encryptionkey@0"
I20260924 03:20:15.072775 2291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "ac155eacfc6346a9bdf11b055c20f0a7"
format_stamp: "Formatted at 2026-09-24 03:20:15 on dist-test-slave-vd0p"
server_key: "38d2ff7e6a8cbaf1bec0c05c1db0151c"
server_key_iv: "24a0a44dc9ae221bcbdcccf645666c28"
server_key_version: "encryptionkey@0"
I20260924 03:20:15.077860 2291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 03:20:15.081252 2306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:15.082551 2291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:15.082671 2291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "ac155eacfc6346a9bdf11b055c20f0a7"
format_stamp: "Formatted at 2026-09-24 03:20:15 on dist-test-slave-vd0p"
server_key: "38d2ff7e6a8cbaf1bec0c05c1db0151c"
server_key_iv: "24a0a44dc9ae221bcbdcccf645666c28"
server_key_version: "encryptionkey@0"
I20260924 03:20:15.082830 2291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:15.103046 2291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:15.107079 2291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:15.107306 2291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:15.108263 2291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:15.109566 2291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:15.109639 2291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:15.109716 2291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:15.109769 2291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:15.140349 2291 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:35923
I20260924 03:20:15.140388 2418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:35923 every 8 connection(s)
I20260924 03:20:15.141600 2291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:20:15.146422 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 2291
I20260924 03:20:15.146615 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:20:15.147063 15246 external_mini_cluster.cc:1428] Setting key 12f8d55440a690db94eaea76379a3f36
Sep 24 03:20:15 dist-test-slave-vd0p krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220015, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.130@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:15.163002 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.144166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:33211 (local address 127.14.227.190:45385)
0924 03:20:15.144353 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:15.144357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:15.145380 (+ 1023us) server_negotiation.cc:407] Connection header received
0924 03:20:15.147355 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:15.147361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:15.147468 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:15.147584 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:15.149243 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.149923 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.151477 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.151890 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.152016 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.155911 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:15.155917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:15.155920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:15.155952 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:15.159591 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:15.160339 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:15.160346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:15.160352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:15.160499 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:15.160975 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:15.160983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:15.160988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:15.161357 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:15.161507 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:15.162298 (+ 791us) server_negotiation.cc:299] Negotiation successful
0924 03:20:15.162565 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:20:15.163733 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.144507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45385 (local address 127.14.227.130:33211)
0924 03:20:15.145215 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:15.145263 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:15.147063 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:15.147805 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:15.147812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:15.148451 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:15.149079 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.149101 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.150107 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.150119 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.151321 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.151334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.152109 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.152116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.152335 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.152997 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:15.153018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:15.155670 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:15.159737 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:15.159744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:15.159758 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:15.160220 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:15.160650 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:15.160654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:15.160656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:15.160854 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:15.161508 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:15.161514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:15.161868 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 03:20:15.162182 (+ 314us) client_negotiation.cc:240] Negotiation successful
0924 03:20:15.162595 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":278,"threads_started":1}
I20260924 03:20:15.165282 2419 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45385
I20260924 03:20:15.165619 2419 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:15.166507 2419 heartbeater.cc:511] Master 127.14.227.190:45385 requested a full tablet report, sending...
I20260924 03:20:15.168093 2096 ts_manager.cc:194] Registered new tserver with Master: ac155eacfc6346a9bdf11b055c20f0a7 (127.14.227.130:35923)
I20260924 03:20:15.168902 2096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.130@KRBTEST.COM'} at 127.14.227.130:33211
WARNING: no policy specified for kudu/127.14.227.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
Entry for principal kudu/127.14.227.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab.
I20260924 03:20:15.205019 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45385
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:15.526489 2424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:15.526762 2424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:15.527010 2424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:15.534307 2424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:15.534427 2424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:15.534611 2424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:20:15.544150 2424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.131.keytab
--principal=kudu/127.14.227.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:15.545424 2424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:15.546778 2424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:15.559316 2430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:15.559322 2432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:15.559275 2429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:15.560154 2424 server_base.cc:1034] running on GCE node
I20260924 03:20:15.560700 2424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:15.561648 2424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:15.562920 2424 hybrid_clock.cc:648] HybridClock initialized: now 1790220015562854 us; error 51 us; skew 500 ppm
I20260924 03:20:15.563246 2424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:15 dist-test-slave-vd0p krb5kdc[2051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220015, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:15.566399 2424 init.cc:377] Logged in from keytab as kudu/127.14.227.131@KRBTEST.COM (short username kudu)
I20260924 03:20:15.567945 2424 webserver.cc:460] Webserver started at http://127.14.227.131:39489/ using document root <none> and password file <none>
I20260924 03:20:15.568600 2424 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:15.568761 2424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:15.569073 2424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:15.571933 2424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "c2ab281155a84ff491deb9df405d270a"
format_stamp: "Formatted at 2026-09-24 03:20:15 on dist-test-slave-vd0p"
server_key: "1fc7fc891c6b733e1f5584cc04f36df1"
server_key_iv: "6be6ace29ef1f41fd5d5c1ad75f14b28"
server_key_version: "encryptionkey@0"
I20260924 03:20:15.572671 2424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "c2ab281155a84ff491deb9df405d270a"
format_stamp: "Formatted at 2026-09-24 03:20:15 on dist-test-slave-vd0p"
server_key: "1fc7fc891c6b733e1f5584cc04f36df1"
server_key_iv: "6be6ace29ef1f41fd5d5c1ad75f14b28"
server_key_version: "encryptionkey@0"
I20260924 03:20:15.577669 2424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:20:15.581101 2439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:15.582377 2424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:15.582551 2424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "c2ab281155a84ff491deb9df405d270a"
format_stamp: "Formatted at 2026-09-24 03:20:15 on dist-test-slave-vd0p"
server_key: "1fc7fc891c6b733e1f5584cc04f36df1"
server_key_iv: "6be6ace29ef1f41fd5d5c1ad75f14b28"
server_key_version: "encryptionkey@0"
I20260924 03:20:15.582765 2424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:15.610517 2424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:15.614770 2424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:15.615073 2424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:15.616009 2424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:15.617268 2424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:15.617365 2424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:15.617511 2424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:15.617583 2424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:15.651153 2424 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:46233
I20260924 03:20:15.651188 2551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:46233 every 8 connection(s)
I20260924 03:20:15.652529 2424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:20:15.657888 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 2424
I20260924 03:20:15.658102 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:20:15.658540 15246 external_mini_cluster.cc:1428] Setting key 35edd6a336415914357faee62ed947db
Sep 24 03:20:15 dist-test-slave-vd0p krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220015, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.131@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:15.673887 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.655102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:52217 (local address 127.14.227.190:45385)
0924 03:20:15.655296 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:15.655301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:15.656521 (+ 1220us) server_negotiation.cc:407] Connection header received
0924 03:20:15.659296 (+ 2775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:15.659302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:15.659414 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:15.659543 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:15.661542 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.662732 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.664197 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.664611 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.664718 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.668026 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:15.668031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:15.668035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:15.668068 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:15.670713 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:15.671340 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:15.671343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:15.671346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:15.671466 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:15.671857 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:15.671862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:15.671867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:15.672242 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:15.672401 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:15.673242 (+ 841us) server_negotiation.cc:299] Negotiation successful
0924 03:20:15.673452 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:20:15.674753 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.655459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45385 (local address 127.14.227.131:52217)
0924 03:20:15.656325 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:15.656377 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:15.658912 (+ 2535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:15.659927 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:15.659935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:15.660530 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:15.661336 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.661358 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.662876 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.662880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.664057 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.664073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.664794 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.664799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.664985 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.665649 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:15.665671 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:15.667838 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:15.670832 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:15.670839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:15.670849 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:15.671238 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:15.671564 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:15.671567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:15.671569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:15.671728 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:15.672380 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:15.672386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:15.672814 (+ 428us) client_negotiation.cc:769] Sending connection context
0924 03:20:15.673096 (+ 282us) client_negotiation.cc:240] Negotiation successful
0924 03:20:15.673565 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":334,"threads_started":1}
I20260924 03:20:15.676085 2552 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45385
I20260924 03:20:15.676389 2552 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:15.677039 2552 heartbeater.cc:511] Master 127.14.227.190:45385 requested a full tablet report, sending...
I20260924 03:20:15.678412 2096 ts_manager.cc:194] Registered new tserver with Master: c2ab281155a84ff491deb9df405d270a (127.14.227.131:46233)
I20260924 03:20:15.679198 2096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.227.131@KRBTEST.COM'} at 127.14.227.131:52217
I20260924 03:20:15.686012 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:20:15.689010 2285 heartbeater.cc:503] Master 127.14.227.190:45385 was elected leader, sending a full tablet report...
I20260924 03:20:15.703375 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.690728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.14.227.190:45385)
0924 03:20:15.690915 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:15.690922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:15.691371 (+ 449us) server_negotiation.cc:407] Connection header received
0924 03:20:15.691497 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:15.691501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:15.691554 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:15.691630 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:15.693072 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.693743 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.695196 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.695594 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.695685 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.698151 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:15.698155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:15.698158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:15.698185 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:15.700771 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:15.701524 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:15.701528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:15.701530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:15.701648 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:15.702059 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:15.702061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:15.702063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:15.702335 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:15.702513 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:15.702883 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 03:20:15.703026 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:20:15.706779 2096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:20:15.713593 2096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:20:15.749161 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.725709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34860)
0924 03:20:15.738809 (+ 13100us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:15.738848 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:15.739034 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:15.740272 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:15.740284 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:15.740330 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:15.740856 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.740874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.743460 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.743466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.745992 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.746007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.748147 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.748153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.748370 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.748415 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 03:20:15.748536 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 03:20:15.748742 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12848,"mutex_wait_us":1,"thread_start_us":17404,"threads_started":1}
I20260924 03:20:15.749408 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.725903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.14.227.129:45103)
0924 03:20:15.737901 (+ 11998us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:15.737915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:15.739261 (+ 1346us) server_negotiation.cc:407] Connection header received
0924 03:20:15.739380 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:15.739390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:15.739697 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:15.739894 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:15.741117 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.743304 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.746173 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.747654 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.747837 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.747998 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:15.748877 (+ 879us) server_negotiation.cc:299] Negotiation successful
0924 03:20:15.749056 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11725,"thread_start_us":12026,"threads_started":1}
I20260924 03:20:15.751518 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.726009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:35923 (local address 127.0.0.1:38594)
0924 03:20:15.743355 (+ 17346us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:15.743377 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:15.743522 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:15.744406 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:15.744411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:15.744434 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:15.744867 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.744879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.746717 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.746722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.748744 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.748754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.750639 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.750645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.750817 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.750864 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:20:15.750965 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 03:20:15.751134 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17234,"mutex_wait_us":31,"thread_start_us":12388,"threads_started":1}
I20260924 03:20:15.751544 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.726303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.14.227.130:35923)
0924 03:20:15.741517 (+ 15214us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:15.741529 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:15.743654 (+ 2125us) server_negotiation.cc:407] Connection header received
0924 03:20:15.743760 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:15.743767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:15.743986 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:15.744174 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:15.745063 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.746558 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.748927 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.750269 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.750370 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.750466 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:15.751043 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 03:20:15.751217 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15000,"thread_start_us":15229,"threads_started":1}
I20260924 03:20:15.753680 2220 tablet_service.cc:1467] Processing CreateTablet for tablet efb626e8575b4b74afaa466b981665ed (DEFAULT_TABLE table=test-table [id=84197830aade43329360d1ce64fba1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:15.755198 2220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb626e8575b4b74afaa466b981665ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:15.755462 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.743696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:46233 (local address 127.0.0.1:52286)
0924 03:20:15.747325 (+ 3629us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:15.747339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:15.747496 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:15.748397 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:15.748401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:15.748424 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:15.748855 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.748868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.750765 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.750771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.752610 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:15.752622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.754586 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.754592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:15.754766 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.754802 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:20:15.754922 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 03:20:15.755053 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3489,"mutex_wait_us":3,"thread_start_us":3345,"threads_started":1}
I20260924 03:20:15.756174 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:15.739645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.14.227.131:46233)
0924 03:20:15.747027 (+ 7382us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:15.747038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:15.747450 (+ 412us) server_negotiation.cc:407] Connection header received
0924 03:20:15.747671 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:15.747683 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:15.747918 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:15.748098 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:15.749062 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.750608 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.752756 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:15.754383 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:15.754595 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:15.754767 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:15.755676 (+ 909us) server_negotiation.cc:299] Negotiation successful
0924 03:20:15.755853 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7201,"thread_start_us":7029,"threads_started":1}
I20260924 03:20:15.755659 2354 tablet_service.cc:1467] Processing CreateTablet for tablet efb626e8575b4b74afaa466b981665ed (DEFAULT_TABLE table=test-table [id=84197830aade43329360d1ce64fba1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:15.756983 2354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb626e8575b4b74afaa466b981665ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:15.759554 2487 tablet_service.cc:1467] Processing CreateTablet for tablet efb626e8575b4b74afaa466b981665ed (DEFAULT_TABLE table=test-table [id=84197830aade43329360d1ce64fba1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:15.760923 2487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efb626e8575b4b74afaa466b981665ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:15.768312 2570 tablet_bootstrap.cc:492] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Bootstrap starting.
I20260924 03:20:15.770648 2571 tablet_bootstrap.cc:492] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: Bootstrap starting.
I20260924 03:20:15.772162 2570 tablet_bootstrap.cc:654] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:15.773455 2570 log.cc:824] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:15.774086 2572 tablet_bootstrap.cc:492] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Bootstrap starting.
I20260924 03:20:15.774371 2571 tablet_bootstrap.cc:654] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:15.775830 2571 log.cc:824] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:15.777019 2572 tablet_bootstrap.cc:654] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:15.778146 2572 log.cc:824] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:15.779068 2571 tablet_bootstrap.cc:492] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: No bootstrap required, opened a new log
I20260924 03:20:15.779322 2570 tablet_bootstrap.cc:492] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: No bootstrap required, opened a new log
I20260924 03:20:15.779349 2571 ts_tablet_manager.cc:1397] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260924 03:20:15.779594 2570 ts_tablet_manager.cc:1397] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260924 03:20:15.781049 2572 tablet_bootstrap.cc:492] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: No bootstrap required, opened a new log
I20260924 03:20:15.781276 2572 ts_tablet_manager.cc:1397] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 03:20:15.786895 2571 raft_consensus.cc:348] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:15.787377 2571 raft_consensus.cc:374] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:15.787513 2571 raft_consensus.cc:729] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfbd5c683d04c4b808ed904e4aef4c8, State: Initialized, Role: FOLLOWER
I20260924 03:20:15.788775 2571 consensus_queue.cc:260] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:15.790781 2571 ts_tablet_manager.cc:1428] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260924 03:20:15.790551 2570 raft_consensus.cc:348] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:15.791039 2570 raft_consensus.cc:374] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:15.791165 2570 raft_consensus.cc:729] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac155eacfc6346a9bdf11b055c20f0a7, State: Initialized, Role: FOLLOWER
I20260924 03:20:15.791718 2570 consensus_queue.cc:260] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:15.791743 2572 raft_consensus.cc:348] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:15.792161 2572 raft_consensus.cc:374] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:15.792325 2572 raft_consensus.cc:729] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ab281155a84ff491deb9df405d270a, State: Initialized, Role: FOLLOWER
I20260924 03:20:15.792804 2419 heartbeater.cc:503] Master 127.14.227.190:45385 was elected leader, sending a full tablet report...
I20260924 03:20:15.792999 2572 consensus_queue.cc:260] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:15.793427 2570 ts_tablet_manager.cc:1428] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260924 03:20:15.794983 2552 heartbeater.cc:503] Master 127.14.227.190:45385 was elected leader, sending a full tablet report...
I20260924 03:20:15.796613 2572 ts_tablet_manager.cc:1428] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260924 03:20:15.893678 2420 tablet.cc:2369] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:20:15.904207 2553 tablet.cc:2369] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:20:15.910084 2286 tablet.cc:2369] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:16.074621 2577 raft_consensus.cc:482] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:20:16.074982 2577 raft_consensus.cc:504] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:16.076704 2577 leader_election.cc:288] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bfbd5c683d04c4b808ed904e4aef4c8 (127.14.227.129:45103), c2ab281155a84ff491deb9df405d270a (127.14.227.131:46233)
I20260924 03:20:16.083508 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.077205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:59293 (local address 127.14.227.129:45103)
0924 03:20:16.077439 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:16.077448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:16.078011 (+ 563us) server_negotiation.cc:407] Connection header received
0924 03:20:16.078192 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:16.078198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:16.078281 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:16.078365 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:16.079014 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.079761 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.081671 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.082520 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.082677 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:16.082780 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:16.083090 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 03:20:16.083232 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:20:16.083544 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.077206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:57231 (local address 127.14.227.131:46233)
0924 03:20:16.077638 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:16.077647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:16.078185 (+ 538us) server_negotiation.cc:407] Connection header received
0924 03:20:16.078360 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:16.078365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:16.078449 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:16.078547 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:16.079167 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.079885 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.081750 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.082542 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.082720 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:16.082809 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:16.083153 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 03:20:16.083296 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260924 03:20:16.083585 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.077207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:46233 (local address 127.14.227.130:57231)
0924 03:20:16.078048 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.078064 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:16.078214 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:16.078577 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:16.078583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:16.078611 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:16.079010 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:16.079022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.080016 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.080019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:16.081539 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:16.081602 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.082794 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.082803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:16.082971 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:16.082992 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 03:20:16.083113 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 03:20:16.083241 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":440,"threads_started":1}
I20260924 03:20:16.083669 2579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.077097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.14.227.130:59293)
0924 03:20:16.077904 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.077919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:16.078054 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:16.078401 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:16.078406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:16.078430 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:16.078865 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:16.078879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.079895 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.079900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:16.081493 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:16.081504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.082744 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.082761 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:16.082932 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:16.082955 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 03:20:16.083038 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 03:20:16.083167 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":84,"spinlock_wait_cycles":25344,"thread_start_us":423,"threads_started":1}
I20260924 03:20:16.084434 2507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb626e8575b4b74afaa466b981665ed" candidate_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ab281155a84ff491deb9df405d270a" is_pre_election: true
I20260924 03:20:16.084474 2240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb626e8575b4b74afaa466b981665ed" candidate_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" is_pre_election: true
I20260924 03:20:16.084910 2507 raft_consensus.cc:2454] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac155eacfc6346a9bdf11b055c20f0a7 in term 0.
I20260924 03:20:16.084962 2240 raft_consensus.cc:2454] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac155eacfc6346a9bdf11b055c20f0a7 in term 0.
I20260924 03:20:16.085578 2310 leader_election.cc:302] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac155eacfc6346a9bdf11b055c20f0a7, c2ab281155a84ff491deb9df405d270a; no voters:
I20260924 03:20:16.086022 2577 raft_consensus.cc:2781] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:20:16.086163 2577 raft_consensus.cc:482] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:20:16.086279 2577 raft_consensus.cc:3037] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:16.088943 2577 raft_consensus.cc:504] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:16.089608 2577 leader_election.cc:288] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [CANDIDATE]: Term 1 election: Requested vote from peers 7bfbd5c683d04c4b808ed904e4aef4c8 (127.14.227.129:45103), c2ab281155a84ff491deb9df405d270a (127.14.227.131:46233)
I20260924 03:20:16.089955 2240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb626e8575b4b74afaa466b981665ed" candidate_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8"
I20260924 03:20:16.090204 2240 raft_consensus.cc:3037] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:16.090312 2507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb626e8575b4b74afaa466b981665ed" candidate_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ab281155a84ff491deb9df405d270a"
I20260924 03:20:16.090528 2507 raft_consensus.cc:3037] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:16.092500 2240 raft_consensus.cc:2454] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac155eacfc6346a9bdf11b055c20f0a7 in term 1.
I20260924 03:20:16.092586 2507 raft_consensus.cc:2454] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac155eacfc6346a9bdf11b055c20f0a7 in term 1.
I20260924 03:20:16.092873 2308 leader_election.cc:302] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bfbd5c683d04c4b808ed904e4aef4c8, ac155eacfc6346a9bdf11b055c20f0a7; no voters:
I20260924 03:20:16.093195 2577 raft_consensus.cc:2781] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:16.093623 2577 raft_consensus.cc:686] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 1 LEADER]: Becoming Leader. State: Replica: ac155eacfc6346a9bdf11b055c20f0a7, State: Running, Role: LEADER
I20260924 03:20:16.094182 2577 consensus_queue.cc:237] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } }
I20260924 03:20:16.098874 2096 catalog_manager.cc:5495] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac155eacfc6346a9bdf11b055c20f0a7 (127.14.227.130). New cstate: current_term: 1 leader_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac155eacfc6346a9bdf11b055c20f0a7" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 35923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 } health_report { overall_health: UNKNOWN } } }
I20260924 03:20:16.188045 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.182654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.14.227.130:35923)
0924 03:20:16.182847 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:16.182853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:16.182916 (+ 63us) server_negotiation.cc:407] Connection header received
0924 03:20:16.183150 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:16.183154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:16.183227 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:16.183375 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:16.183745 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.184414 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.185773 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.186334 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.186479 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:16.186947 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:16.187479 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:16.187654 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:16.187701 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 03:20:16.187777 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:20:16.196117 2240 raft_consensus.cc:1261] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 1 FOLLOWER]: Refusing update from remote peer ac155eacfc6346a9bdf11b055c20f0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:16.196188 2507 raft_consensus.cc:1261] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 1 FOLLOWER]: Refusing update from remote peer ac155eacfc6346a9bdf11b055c20f0a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:16.197054 2577 consensus_queue.cc:1035] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfbd5c683d04c4b808ed904e4aef4c8" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 45103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:16.197415 2581 consensus_queue.cc:1035] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ab281155a84ff491deb9df405d270a" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:16.208989 2584 mvcc.cc:205] Tried to move back new op lower bound from 7332741186328842240 to 7332741185924812800. Current Snapshot: MvccSnapshot[applied={T|T < 7332741186328842240}]
I20260924 03:20:16.212695 2586 mvcc.cc:205] Tried to move back new op lower bound from 7332741186328842240 to 7332741185924812800. Current Snapshot: MvccSnapshot[applied={T|T < 7332741186328842240}]
I20260924 03:20:16.222352 2095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:16.222643 2095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:16.226532 2095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9745222d8df74c85a81bf2aaa1c02513: Sending DeleteTablet for 3 replicas of tablet efb626e8575b4b74afaa466b981665ed
I20260924 03:20:16.227751 2487 tablet_service.cc:1514] Processing DeleteTablet for tablet efb626e8575b4b74afaa466b981665ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:16 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:52286
I20260924 03:20:16.227751 2220 tablet_service.cc:1514] Processing DeleteTablet for tablet efb626e8575b4b74afaa466b981665ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:16 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:34860
I20260924 03:20:16.227836 2354 tablet_service.cc:1514] Processing DeleteTablet for tablet efb626e8575b4b74afaa466b981665ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:16 UTC) from {username='kudu', principal='kudu/127.14.227.190@KRBTEST.COM'} at 127.0.0.1:38594
I20260924 03:20:16.229887 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 2157
I20260924 03:20:16.238607 2595 tablet_replica.cc:330] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: stopping tablet replica
I20260924 03:20:16.239842 2595 raft_consensus.cc:2229] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:20:16.240710 2593 tablet_replica.cc:330] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: stopping tablet replica
I20260924 03:20:16.241156 2595 raft_consensus.cc:2258] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:16.241212 2593 raft_consensus.cc:2229] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:16.241987 2593 raft_consensus.cc:2258] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:16.244401 2593 ts_tablet_manager.cc:1905] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:16.244407 2595 ts_tablet_manager.cc:1905] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 03:20:16 dist-test-slave-vd0p krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.129@KRBTEST.COM
I20260924 03:20:16.250958 2594 tablet_replica.cc:330] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: stopping tablet replica
I20260924 03:20:16.251443 2594 raft_consensus.cc:2229] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:16.252296 2594 raft_consensus.cc:2258] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:16.252511 2595 ts_tablet_manager.cc:1918] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:16.252933 2595 log.cc:1196] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/efb626e8575b4b74afaa466b981665ed
I20260924 03:20:16.253688 2595 ts_tablet_manager.cc:1939] T efb626e8575b4b74afaa466b981665ed P ac155eacfc6346a9bdf11b055c20f0a7: Deleting consensus metadata
I20260924 03:20:16.254470 2594 ts_tablet_manager.cc:1905] T efb626e8575b4b74afaa466b981665ed P 7bfbd5c683d04c4b808ed904e4aef4c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:16.255560 2083 catalog_manager.cc:4841] TS ac155eacfc6346a9bdf11b055c20f0a7 (127.14.227.130:35923): tablet efb626e8575b4b74afaa466b981665ed (table test-table [id=84197830aade43329360d1ce64fba1a8]) successfully deleted
I20260924 03:20:16.258054 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.232830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.14.227.129:45103)
0924 03:20:16.233060 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:16.233067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:16.237623 (+ 4556us) server_negotiation.cc:407] Connection header received
0924 03:20:16.237841 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:16.237847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:16.237940 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:16.238084 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:16.240075 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.241226 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.243112 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.243760 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.243861 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:16.249484 (+ 5623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:16.249502 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:16.249519 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:16.249561 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:16.253781 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:16.254739 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:16.254749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:16.254752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:16.254947 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:16.256200 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:16.256206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:16.256210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:16.256695 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:16.256988 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:16.257357 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 03:20:16.257584 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:20:16.259065 2280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:16.259935 2593 ts_tablet_manager.cc:1918] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:16.260174 2593 log.cc:1196] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/efb626e8575b4b74afaa466b981665ed
I20260924 03:20:16.260789 2593 ts_tablet_manager.cc:1939] T efb626e8575b4b74afaa466b981665ed P c2ab281155a84ff491deb9df405d270a: Deleting consensus metadata
I20260924 03:20:16.262701 2083 catalog_manager.cc:4841] TS c2ab281155a84ff491deb9df405d270a (127.14.227.131:46233): tablet efb626e8575b4b74afaa466b981665ed (table test-table [id=84197830aade43329360d1ce64fba1a8]) successfully deleted
I20260924 03:20:16.353034 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 2157
W20260924 03:20:16.370599 2081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.227.129:45103 (error 108)
I20260924 03:20:16.370774 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 2291
I20260924 03:20:16.372714 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.371920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34878)
0924 03:20:16.372337 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.372486 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":463,"threads_started":1}
W20260924 03:20:16.372974 2081 catalog_manager.cc:4584] TS 7bfbd5c683d04c4b808ed904e4aef4c8 (127.14.227.129:45103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efb626e8575b4b74afaa466b981665ed: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Sep 24 03:20:16 dist-test-slave-vd0p krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.130@KRBTEST.COM
I20260924 03:20:16.389940 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.371834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.14.227.130:35923)
0924 03:20:16.372000 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:16.372004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:16.372215 (+ 211us) server_negotiation.cc:407] Connection header received
0924 03:20:16.372428 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:16.372436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:16.372526 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:16.372646 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:16.374980 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.375784 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.377710 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.378302 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.378406 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:16.383856 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:16.383872 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:16.383887 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:16.383927 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:16.387318 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:16.387956 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:16.387964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:16.387968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:16.388118 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:16.388478 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:16.388484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:16.388487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:16.388829 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:16.389166 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:16.389362 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 03:20:16.389600 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:20:16.390717 2414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:16.426535 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.426092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34884)
0924 03:20:16.426282 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.426378 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:20:16.427551 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.427207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34896)
0924 03:20:16.427358 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.427425 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 03:20:16.487960 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.487510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34900)
0924 03:20:16.487716 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.487802 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 03:20:16.488994 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.488615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34910)
0924 03:20:16.488784 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.488851 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 03:20:16.502746 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 2291
I20260924 03:20:16.521185 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 2424
Sep 24 03:20:16 dist-test-slave-vd0p krb5kdc[2051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.131@KRBTEST.COM
I20260924 03:20:16.536268 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.522089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.14.227.131:46233)
0924 03:20:16.522258 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:16.522263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:16.522305 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:20:16.522480 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:16.522484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:16.522547 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:16.522642 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:16.523884 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.524735 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.526094 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:16.526510 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:16.526607 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:16.530868 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:16.530882 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:16.530899 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:16.530929 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:16.533602 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:16.534527 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:16.534534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:16.534537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:16.534668 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:16.534979 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:16.534982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:16.534984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:16.535305 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:16.535514 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:16.535736 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 03:20:16.535915 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:20:16.536903 2547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:16.569479 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.569046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34912)
0924 03:20:16.569258 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.569338 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 03:20:16.570622 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.570176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34928)
0924 03:20:16.570406 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.570474 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 03:20:16.624246 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 2424
I20260924 03:20:16.640403 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 2064
I20260924 03:20:16.641548 2126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:16.751261 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:16.750474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:45103 (local address 127.0.0.1:34936)
0924 03:20:16.750702 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:16.750789 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.129:45103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 03:20:16.751354 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 2064
2026-09-24T03:20:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4498 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T03:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:16Z Disabled control of system clock
I20260924 03:20:16.822660 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:45939
--webserver_interface=127.14.227.190
--webserver_port=0
--builtin_ntp_servers=127.14.227.148:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:45939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:20:17.143527 2606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:17.143810 2606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:17.143888 2606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:17.150898 2606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:17.150996 2606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:17.151041 2606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:17.151096 2606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:17.151139 2606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:17.159884 2606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:45939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:17.161063 2606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:17.162741 2606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:17.173915 2612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:17.173983 2614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:17.174017 2611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:17.174973 2606 server_base.cc:1034] running on GCE node
I20260924 03:20:17.175743 2606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:17.177096 2606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:17.178294 2606 hybrid_clock.cc:648] HybridClock initialized: now 1790220017178262 us; error 31 us; skew 500 ppm
I20260924 03:20:17.178620 2606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:17.180795 2606 webserver.cc:460] Webserver started at http://127.14.227.190:39567/ using document root <none> and password file <none>
I20260924 03:20:17.181447 2606 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:17.181535 2606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:17.181777 2606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:17.184607 2606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "a6beda5d89ce464aae1ca934f83215eb"
format_stamp: "Formatted at 2026-09-24 03:20:17 on dist-test-slave-vd0p"
server_key: "9adf8877fb1a3b1b95ce49c2541de013"
server_key_iv: "3b6bf192ac15835fc750932267eb5638"
server_key_version: "encryptionkey@0"
I20260924 03:20:17.185328 2606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "a6beda5d89ce464aae1ca934f83215eb"
format_stamp: "Formatted at 2026-09-24 03:20:17 on dist-test-slave-vd0p"
server_key: "9adf8877fb1a3b1b95ce49c2541de013"
server_key_iv: "3b6bf192ac15835fc750932267eb5638"
server_key_version: "encryptionkey@0"
I20260924 03:20:17.190145 2606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 03:20:17.193594 2620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:17.194943 2606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:17.195132 2606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "a6beda5d89ce464aae1ca934f83215eb"
format_stamp: "Formatted at 2026-09-24 03:20:17 on dist-test-slave-vd0p"
server_key: "9adf8877fb1a3b1b95ce49c2541de013"
server_key_iv: "3b6bf192ac15835fc750932267eb5638"
server_key_version: "encryptionkey@0"
I20260924 03:20:17.195313 2606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:17.209878 2606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:17.210726 2606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:17.211006 2606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:17.228969 2606 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:45939
I20260924 03:20:17.229034 2671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:45939 every 8 connection(s)
I20260924 03:20:17.230285 2606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:17.233588 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:17.234298 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 2606
I20260924 03:20:17.234481 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:17.234946 15246 external_mini_cluster.cc:1428] Setting key b0f5a25dd1301131bfe463e87e37ca39
I20260924 03:20:17.245255 2672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb: Bootstrap starting.
I20260924 03:20:17.247763 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:17.236749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.14.227.190:45939)
0924 03:20:17.237672 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:17.237689 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:17.237726 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:20:17.239686 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:17.239706 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:17.240130 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:17.240498 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:17.241014 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:17.242337 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:17.243634 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:17.244247 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:17.244434 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:17.245599 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:17.245607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:17.245619 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:17.246183 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:17.246264 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:17.246284 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:17.246404 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:17.246646 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 03:20:17.246927 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":329,"threads_started":1}
I20260924 03:20:17.248391 2672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:17.249387 2672 log.cc:824] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:17.252004 2672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb: No bootstrap required, opened a new log
I20260924 03:20:17.259002 2672 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6beda5d89ce464aae1ca934f83215eb" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45939 } }
I20260924 03:20:17.259362 2672 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:17.259490 2672 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6beda5d89ce464aae1ca934f83215eb, State: Initialized, Role: FOLLOWER
I20260924 03:20:17.260039 2672 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6beda5d89ce464aae1ca934f83215eb" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45939 } }
I20260924 03:20:17.260272 2672 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:17.260444 2672 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:17.260608 2672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:17.262864 2672 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6beda5d89ce464aae1ca934f83215eb" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45939 } }
I20260924 03:20:17.263337 2672 leader_election.cc:302] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6beda5d89ce464aae1ca934f83215eb; no voters:
I20260924 03:20:17.263888 2672 leader_election.cc:288] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:17.264043 2677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:17.264750 2677 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [term 1 LEADER]: Becoming Leader. State: Replica: a6beda5d89ce464aae1ca934f83215eb, State: Running, Role: LEADER
I20260924 03:20:17.265213 2677 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6beda5d89ce464aae1ca934f83215eb" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45939 } }
I20260924 03:20:17.265918 2672 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:17.268227 2679 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6beda5d89ce464aae1ca934f83215eb. Latest consensus state: current_term: 1 leader_uuid: "a6beda5d89ce464aae1ca934f83215eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6beda5d89ce464aae1ca934f83215eb" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45939 } } }
I20260924 03:20:17.268682 2679 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:17.268353 2678 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6beda5d89ce464aae1ca934f83215eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6beda5d89ce464aae1ca934f83215eb" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 45939 } } }
I20260924 03:20:17.268949 2678 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:17.270716 2683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:17.275961 2683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:17.284747 2683 catalog_manager.cc:1347] Generated new cluster ID: ed3717e2406442d6a108c657b92e9771
I20260924 03:20:17.284842 2683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:17.299513 2683 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:17.300747 2683 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:17.308975 2683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb: Generated new TSK 0
I20260924 03:20:17.309746 2683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:17.322515 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.129:0
--local_ip_for_outbound_sockets=127.14.227.129
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45939
--builtin_ntp_servers=127.14.227.148:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:20:17.648343 2696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:17.648641 2696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:17.648799 2696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:17.656312 2696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:17.656435 2696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:17.656607 2696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.129
I20260924 03:20:17.666221 2696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.227.129
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.129
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:17.667599 2696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:17.669288 2696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:17.682220 2702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:17.682297 2701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:17.682308 2704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:17.683391 2696 server_base.cc:1034] running on GCE node
I20260924 03:20:17.684048 2696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:17.684975 2696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:17.686175 2696 hybrid_clock.cc:648] HybridClock initialized: now 1790220017686143 us; error 34 us; skew 500 ppm
I20260924 03:20:17.686440 2696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:17.688735 2696 webserver.cc:460] Webserver started at http://127.14.227.129:35297/ using document root <none> and password file <none>
I20260924 03:20:17.689378 2696 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:17.689458 2696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:17.689697 2696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:17.692454 2696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/data/instance:
uuid: "8b079cf51b2b41b3855132ed00448fd2"
format_stamp: "Formatted at 2026-09-24 03:20:17 on dist-test-slave-vd0p"
server_key: "34aaadea1cf0b91b87a76871fca7707b"
server_key_iv: "c20aed28d3b3e96b24d806a32d0bad24"
server_key_version: "encryptionkey@0"
I20260924 03:20:17.693200 2696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance:
uuid: "8b079cf51b2b41b3855132ed00448fd2"
format_stamp: "Formatted at 2026-09-24 03:20:17 on dist-test-slave-vd0p"
server_key: "34aaadea1cf0b91b87a76871fca7707b"
server_key_iv: "c20aed28d3b3e96b24d806a32d0bad24"
server_key_version: "encryptionkey@0"
I20260924 03:20:17.698297 2696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 03:20:17.702030 2710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:17.703296 2696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:20:17.703471 2696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/wal
uuid: "8b079cf51b2b41b3855132ed00448fd2"
format_stamp: "Formatted at 2026-09-24 03:20:17 on dist-test-slave-vd0p"
server_key: "34aaadea1cf0b91b87a76871fca7707b"
server_key_iv: "c20aed28d3b3e96b24d806a32d0bad24"
server_key_version: "encryptionkey@0"
I20260924 03:20:17.703676 2696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:17.728127 2696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:17.728937 2696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:17.729233 2696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:17.730221 2696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:17.731638 2696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:17.731760 2696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:17.731891 2696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:17.731984 2696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:17.764309 2696 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.129:39741
I20260924 03:20:17.764392 2822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.129:39741 every 8 connection(s)
I20260924 03:20:17.765607 2696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/data/info.pb
I20260924 03:20:17.775810 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 2696
I20260924 03:20:17.776027 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/wal/instance
I20260924 03:20:17.776530 15246 external_mini_cluster.cc:1428] Setting key 1e8087c036da9331ad8d425bd68d5a51
I20260924 03:20:17.778007 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:17.768288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.129:36131 (local address 127.14.227.190:45939)
0924 03:20:17.768744 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:17.768750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:17.770147 (+ 1397us) server_negotiation.cc:407] Connection header received
0924 03:20:17.771864 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:17.771869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:17.771931 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:17.772024 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:17.772964 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:17.773635 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:17.775106 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:17.775582 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:17.775711 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:17.777111 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:17.777116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:17.777119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:17.777135 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:17.777173 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:17.777185 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:17.777288 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:17.777528 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 03:20:17.777638 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260924 03:20:17.778914 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:17.768859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45939 (local address 127.14.227.129:36131)
0924 03:20:17.769944 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:17.770035 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:17.771602 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:17.772205 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:17.772212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:17.772300 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:17.772828 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:17.772844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:17.773754 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:17.773758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:17.774979 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:17.774991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:17.775790 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:17.775795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:17.776051 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:17.776807 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:17.776810 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:17.776845 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:17.776915 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:17.777290 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:17.777305 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:17.777314 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:20:17.777504 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 03:20:17.777840 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":335,"threads_started":1}
I20260924 03:20:17.779361 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.130:0
--local_ip_for_outbound_sockets=127.14.227.130
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45939
--builtin_ntp_servers=127.14.227.148:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:20:17.780356 2823 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45939
I20260924 03:20:17.780665 2823 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:17.781493 2823 heartbeater.cc:511] Master 127.14.227.190:45939 requested a full tablet report, sending...
I20260924 03:20:17.783486 2637 ts_manager.cc:194] Registered new tserver with Master: 8b079cf51b2b41b3855132ed00448fd2 (127.14.227.129:39741)
I20260924 03:20:17.784883 2637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.129:36131
W20260924 03:20:18.095247 2827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:18.095505 2827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:18.095643 2827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:18.102631 2827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:18.102710 2827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:18.102893 2827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.130
I20260924 03:20:18.111513 2827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.227.130
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.130
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:18.112694 2827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:18.114390 2827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:18.126466 2832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:18.126499 2833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:18.126547 2835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:18.127583 2827 server_base.cc:1034] running on GCE node
I20260924 03:20:18.128113 2827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:18.128961 2827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:18.130148 2827 hybrid_clock.cc:648] HybridClock initialized: now 1790220018130114 us; error 34 us; skew 500 ppm
I20260924 03:20:18.130402 2827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:18.132519 2827 webserver.cc:460] Webserver started at http://127.14.227.130:35765/ using document root <none> and password file <none>
I20260924 03:20:18.133114 2827 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:18.133194 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:18.133425 2827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:18.136065 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/data/instance:
uuid: "96fd22ad7a854cc2926b761f91883c6f"
format_stamp: "Formatted at 2026-09-24 03:20:18 on dist-test-slave-vd0p"
server_key: "dc433998b839d11622daea1d11c88b17"
server_key_iv: "ea3ef779a48e90a28a3351ef06212a15"
server_key_version: "encryptionkey@0"
I20260924 03:20:18.136773 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance:
uuid: "96fd22ad7a854cc2926b761f91883c6f"
format_stamp: "Formatted at 2026-09-24 03:20:18 on dist-test-slave-vd0p"
server_key: "dc433998b839d11622daea1d11c88b17"
server_key_iv: "ea3ef779a48e90a28a3351ef06212a15"
server_key_version: "encryptionkey@0"
I20260924 03:20:18.141436 2827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:20:18.144758 2841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:18.146013 2827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 03:20:18.146155 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/wal
uuid: "96fd22ad7a854cc2926b761f91883c6f"
format_stamp: "Formatted at 2026-09-24 03:20:18 on dist-test-slave-vd0p"
server_key: "dc433998b839d11622daea1d11c88b17"
server_key_iv: "ea3ef779a48e90a28a3351ef06212a15"
server_key_version: "encryptionkey@0"
I20260924 03:20:18.146314 2827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:18.159612 2827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:18.160261 2827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:18.160503 2827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:18.161314 2827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:18.162672 2827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:18.162753 2827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:18.162827 2827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:18.162878 2827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:18.194617 2827 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.130:46745
I20260924 03:20:18.194667 2953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.130:46745 every 8 connection(s)
I20260924 03:20:18.195917 2827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/data/info.pb
I20260924 03:20:18.203059 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 2827
I20260924 03:20:18.203276 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/wal/instance
I20260924 03:20:18.203856 15246 external_mini_cluster.cc:1428] Setting key f66913b29213fb3c08f0c0373be2a13d
I20260924 03:20:18.206811 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.227.131:0
--local_ip_for_outbound_sockets=127.14.227.131
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45939
--builtin_ntp_servers=127.14.227.148:39705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:20:18.209036 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.198570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:41007 (local address 127.14.227.190:45939)
0924 03:20:18.198787 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:18.198793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:18.200228 (+ 1435us) server_negotiation.cc:407] Connection header received
0924 03:20:18.201901 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:18.201906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:18.202024 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:18.202108 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:18.203177 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.204253 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.206015 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.206628 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.206764 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.208228 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:18.208235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:18.208239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:18.208263 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:18.208292 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:18.208303 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:18.208396 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:18.208616 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 03:20:18.208705 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:20:18.210135 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.199113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45939 (local address 127.14.227.130:41007)
0924 03:20:18.200085 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:18.200122 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:18.201653 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:18.202306 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:18.202313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:18.202403 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:18.202990 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.203006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.204416 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.204421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.205861 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.205873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.206850 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.206856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.207171 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.207888 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:18.207896 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:18.207934 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:18.208003 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:18.208396 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:18.208408 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:18.208417 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 03:20:18.208606 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 03:20:18.208967 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":263,"threads_started":1}
I20260924 03:20:18.211551 2954 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45939
I20260924 03:20:18.211861 2954 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:18.212592 2954 heartbeater.cc:511] Master 127.14.227.190:45939 requested a full tablet report, sending...
I20260924 03:20:18.213920 2637 ts_manager.cc:194] Registered new tserver with Master: 96fd22ad7a854cc2926b761f91883c6f (127.14.227.130:46745)
I20260924 03:20:18.214986 2637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.130:41007
W20260924 03:20:18.553880 2957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:18.554265 2957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:18.554472 2957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:18.562831 2957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:18.562939 2957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:18.563153 2957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.227.131
I20260924 03:20:18.573787 2957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.227.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.227.131
--webserver_port=0
--tserver_master_addrs=127.14.227.190:45939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.227.131
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:18.575260 2957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:18.577240 2957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:18.591508 2962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:18.591511 2963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:18.591536 2965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:18.592168 2957 server_base.cc:1034] running on GCE node
I20260924 03:20:18.592820 2957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:18.593779 2957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:18.595006 2957 hybrid_clock.cc:648] HybridClock initialized: now 1790220018594954 us; error 65 us; skew 500 ppm
I20260924 03:20:18.595335 2957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 03:20:18.597663 2957 webserver.cc:460] Webserver started at http://127.14.227.131:43045/ using document root <none> and password file <none>
I20260924 03:20:18.598465 2957 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:18.598618 2957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:18.598943 2957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:18.601888 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/data/instance:
uuid: "3440b59e10804420bc376a68cf05e37f"
format_stamp: "Formatted at 2026-09-24 03:20:18 on dist-test-slave-vd0p"
server_key: "03e4835620acd738f916e6202281e1ab"
server_key_iv: "3b28019a3ee97663bc52320a7bffae78"
server_key_version: "encryptionkey@0"
I20260924 03:20:18.602733 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance:
uuid: "3440b59e10804420bc376a68cf05e37f"
format_stamp: "Formatted at 2026-09-24 03:20:18 on dist-test-slave-vd0p"
server_key: "03e4835620acd738f916e6202281e1ab"
server_key_iv: "3b28019a3ee97663bc52320a7bffae78"
server_key_version: "encryptionkey@0"
I20260924 03:20:18.607730 2957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:20:18.611096 2971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:18.612362 2957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:20:18.612484 2957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/wal
uuid: "3440b59e10804420bc376a68cf05e37f"
format_stamp: "Formatted at 2026-09-24 03:20:18 on dist-test-slave-vd0p"
server_key: "03e4835620acd738f916e6202281e1ab"
server_key_iv: "3b28019a3ee97663bc52320a7bffae78"
server_key_version: "encryptionkey@0"
I20260924 03:20:18.612651 2957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:18.625770 2957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:18.626639 2957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:18.626889 2957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:18.627748 2957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:20:18.629087 2957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:20:18.629169 2957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:18.629259 2957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:20:18.629313 2957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:18.662289 2957 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.131:37117
I20260924 03:20:18.662325 3083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.131:37117 every 8 connection(s)
I20260924 03:20:18.663596 2957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/data/info.pb
I20260924 03:20:18.672690 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 2957
I20260924 03:20:18.672880 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/wal/instance
I20260924 03:20:18.673312 15246 external_mini_cluster.cc:1428] Setting key 29cea97c0a86fd12d33ccc0a08abcb81
I20260924 03:20:18.676123 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.666138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.131:42997 (local address 127.14.227.190:45939)
0924 03:20:18.666329 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:18.666333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:18.667688 (+ 1355us) server_negotiation.cc:407] Connection header received
0924 03:20:18.669351 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:18.669355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:18.669421 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:18.669509 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:18.670564 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.671447 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.673124 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.673584 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.673721 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.675223 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:18.675228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:18.675231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:18.675268 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:18.675313 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:18.675332 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:18.675436 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:18.675647 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 03:20:18.675760 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:20:18.676970 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.666619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:45939 (local address 127.14.227.131:42997)
0924 03:20:18.667518 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:18.667557 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:18.669118 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:18.669710 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:18.669718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:18.669800 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:18.670420 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.670437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.671596 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.671600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.672976 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.672996 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.673796 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.673803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.674113 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.674869 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:20:18.674873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:18.674921 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:20:18.675029 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:18.675440 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:18.675449 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:18.675455 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:20:18.675594 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 03:20:18.675875 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":290,"threads_started":1}
I20260924 03:20:18.678333 3084 heartbeater.cc:347] Connected to a master server at 127.14.227.190:45939
I20260924 03:20:18.678617 3084 heartbeater.cc:464] Registering TS with master...
I20260924 03:20:18.679307 3084 heartbeater.cc:511] Master 127.14.227.190:45939 requested a full tablet report, sending...
I20260924 03:20:18.680598 2637 ts_manager.cc:194] Registered new tserver with Master: 3440b59e10804420bc376a68cf05e37f (127.14.227.131:37117)
I20260924 03:20:18.681372 2637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.227.131:42997
I20260924 03:20:18.688442 15246 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:20:18.697902 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.692480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.14.227.190:45939)
0924 03:20:18.692676 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:18.692684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:18.693059 (+ 375us) server_negotiation.cc:407] Connection header received
0924 03:20:18.693190 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:18.693194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:18.693260 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:18.693337 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:18.693778 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.694607 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.695895 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.696270 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.696357 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.697231 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:18.697235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:18.697238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:18.697253 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:18.697281 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:18.697291 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:18.697357 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:18.697469 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 03:20:18.697534 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:20:18.701319 2637 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:20:18.708148 2637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:20:18.728965 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.719463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:37117 (local address 127.0.0.1:33808)
0924 03:20:18.720233 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:18.720274 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:18.720444 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:18.721484 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:18.721497 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:18.721541 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:18.722067 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.722084 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.724343 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.724349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.726613 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.726627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.728042 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.728061 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.728270 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.728329 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 03:20:18.728463 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 03:20:18.728647 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":337,"threads_started":1}
I20260924 03:20:18.729017 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.719756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.14.227.131:37117)
0924 03:20:18.720526 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:18.720535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:18.720569 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:20:18.720663 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:18.720672 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:18.720956 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:18.721145 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:18.722311 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.724201 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.726829 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.727835 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.728043 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.728263 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:18.728504 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 03:20:18.728674 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":441,"threads_started":1}
I20260924 03:20:18.730525 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.721182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.130:46745 (local address 127.0.0.1:38690)
0924 03:20:18.722673 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:18.722689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:18.722795 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:18.723751 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:18.723755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:18.723780 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:18.724150 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.724161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.725927 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.725931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.728115 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.728130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.729989 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.729995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.730128 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.730150 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 03:20:18.730223 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:20:18.730302 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"thread_start_us":1250,"threads_started":1}
I20260924 03:20:18.730799 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.721640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.14.227.130:46745)
0924 03:20:18.723289 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:18.723298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:18.723328 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:20:18.723413 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:18.723417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:18.723570 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:18.723751 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:18.724331 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.725775 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.728351 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.729783 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.729937 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.730153 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:18.730284 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:20:18.730447 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1488,"thread_start_us":1365,"threads_started":1}
I20260924 03:20:18.733793 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.720631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39741 (local address 127.0.0.1:56000)
0924 03:20:18.721207 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:18.721230 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:18.721363 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:18.724875 (+ 3512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:18.724882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:18.724913 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:18.725300 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.725314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.727186 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.727191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.731129 (+ 3938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.731141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.733051 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.733062 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.733264 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.733296 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 03:20:18.733393 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 03:20:18.733513 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":252,"threads_started":1}
I20260924 03:20:18.733228 3019 tablet_service.cc:1467] Processing CreateTablet for tablet c7e51ed93e024b6ca3f3e6d055a5df03 (DEFAULT_TABLE table=test-table [id=6dbe712c62144ab2b98b95fd17b7174e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:18.734160 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.720941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.14.227.129:39741)
0924 03:20:18.724341 (+ 3400us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:18.724349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:18.724376 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:20:18.724474 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:18.724481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:18.724689 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:18.724869 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:18.725517 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.727038 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.731306 (+ 4268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.732578 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.732762 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.732948 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:18.733453 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 03:20:18.733620 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3226,"thread_start_us":3091,"threads_started":1}
I20260924 03:20:18.734758 3019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e51ed93e024b6ca3f3e6d055a5df03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:18.734426 2889 tablet_service.cc:1467] Processing CreateTablet for tablet c7e51ed93e024b6ca3f3e6d055a5df03 (DEFAULT_TABLE table=test-table [id=6dbe712c62144ab2b98b95fd17b7174e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:18.735765 2889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e51ed93e024b6ca3f3e6d055a5df03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:18.737421 2758 tablet_service.cc:1467] Processing CreateTablet for tablet c7e51ed93e024b6ca3f3e6d055a5df03 (DEFAULT_TABLE table=test-table [id=6dbe712c62144ab2b98b95fd17b7174e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:20:18.738664 2758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e51ed93e024b6ca3f3e6d055a5df03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:18.745492 3102 tablet_bootstrap.cc:492] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Bootstrap starting.
I20260924 03:20:18.748446 3102 tablet_bootstrap.cc:654] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:18.749816 3102 log.cc:824] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:18.749990 3103 tablet_bootstrap.cc:492] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Bootstrap starting.
I20260924 03:20:18.752235 3105 tablet_bootstrap.cc:492] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Bootstrap starting.
I20260924 03:20:18.753142 3103 tablet_bootstrap.cc:654] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:18.753191 3102 tablet_bootstrap.cc:492] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: No bootstrap required, opened a new log
I20260924 03:20:18.753408 3102 ts_tablet_manager.cc:1397] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 03:20:18.754271 3103 log.cc:824] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:18.755108 3105 tablet_bootstrap.cc:654] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:18.756254 3105 log.cc:824] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:18.757187 3103 tablet_bootstrap.cc:492] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: No bootstrap required, opened a new log
I20260924 03:20:18.757426 3103 ts_tablet_manager.cc:1397] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 03:20:18.758746 3105 tablet_bootstrap.cc:492] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: No bootstrap required, opened a new log
I20260924 03:20:18.759022 3105 ts_tablet_manager.cc:1397] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260924 03:20:18.763234 3102 raft_consensus.cc:348] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:18.763662 3102 raft_consensus.cc:374] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:18.763746 3102 raft_consensus.cc:729] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3440b59e10804420bc376a68cf05e37f, State: Initialized, Role: FOLLOWER
I20260924 03:20:18.764508 3102 consensus_queue.cc:260] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:18.767243 3105 raft_consensus.cc:348] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:18.767758 3105 raft_consensus.cc:374] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:18.767906 3105 raft_consensus.cc:729] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b079cf51b2b41b3855132ed00448fd2, State: Initialized, Role: FOLLOWER
I20260924 03:20:18.768405 3103 raft_consensus.cc:348] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:18.768586 3105 consensus_queue.cc:260] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:18.768882 3103 raft_consensus.cc:374] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:18.768999 3103 raft_consensus.cc:729] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fd22ad7a854cc2926b761f91883c6f, State: Initialized, Role: FOLLOWER
I20260924 03:20:18.769745 3103 consensus_queue.cc:260] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:18.772925 3084 heartbeater.cc:503] Master 127.14.227.190:45939 was elected leader, sending a full tablet report...
I20260924 03:20:18.773638 3102 ts_tablet_manager.cc:1428] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Time spent starting tablet: real 0.020s user 0.012s sys 0.007s
I20260924 03:20:18.774480 2954 heartbeater.cc:503] Master 127.14.227.190:45939 was elected leader, sending a full tablet report...
I20260924 03:20:18.776418 3103 ts_tablet_manager.cc:1428] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20260924 03:20:18.778584 2823 heartbeater.cc:503] Master 127.14.227.190:45939 was elected leader, sending a full tablet report...
I20260924 03:20:18.779137 3105 ts_tablet_manager.cc:1428] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Time spent starting tablet: real 0.020s user 0.007s sys 0.012s
W20260924 03:20:18.915442 3085 tablet.cc:2369] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:20:18.948230 2955 tablet.cc:2369] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:18.976341 3110 raft_consensus.cc:482] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:20:18.976742 3110 raft_consensus.cc:504] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:18.978579 3110 leader_election.cc:288] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3440b59e10804420bc376a68cf05e37f (127.14.227.131:37117), 8b079cf51b2b41b3855132ed00448fd2 (127.14.227.129:39741)
I20260924 03:20:18.985613 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.978830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.131:37117 (local address 127.14.227.130:59105)
0924 03:20:18.979503 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:18.979522 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:18.979675 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:18.980167 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:18.980170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:18.980196 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:18.980645 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.980662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.981933 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.981938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.983481 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.983493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.984865 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.984873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.985055 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.985091 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:20:18.985184 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:20:18.985305 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":3,"thread_start_us":340,"threads_started":1}
I20260924 03:20:18.985657 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.979082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:59105 (local address 127.14.227.131:37117)
0924 03:20:18.979330 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:18.979337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:18.979633 (+ 296us) server_negotiation.cc:407] Connection header received
0924 03:20:18.979858 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:18.979864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:18.979940 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:18.980012 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:18.980799 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.981778 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.983662 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.984674 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.984806 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.984855 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:18.985226 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 03:20:18.985360 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 03:20:18.986158 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.979727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.227.130:35861 (local address 127.14.227.129:39741)
0924 03:20:18.979932 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:18.979938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:18.980319 (+ 381us) server_negotiation.cc:407] Connection header received
0924 03:20:18.980483 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:18.980488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:18.980579 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:18.980683 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:20:18.981362 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.982156 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.984187 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.984981 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.985115 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.985192 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:18.985642 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 03:20:18.985786 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:20:18.986634 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:18.979547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.129:39741 (local address 127.14.227.130:35861)
0924 03:20:18.980200 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:18.980216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:18.980345 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:18.980734 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:18.980742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:18.980770 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:20:18.981179 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.981193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.982290 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.982294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.983994 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:18.984011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:18.985186 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:18.985193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:18.985377 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:18.985412 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 03:20:18.986113 (+ 701us) client_negotiation.cc:240] Negotiation successful
0924 03:20:18.986329 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":42,"thread_start_us":332,"threads_started":1}
I20260924 03:20:18.987125 3039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e51ed93e024b6ca3f3e6d055a5df03" candidate_uuid: "96fd22ad7a854cc2926b761f91883c6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3440b59e10804420bc376a68cf05e37f" is_pre_election: true
I20260924 03:20:18.987730 3039 raft_consensus.cc:2454] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fd22ad7a854cc2926b761f91883c6f in term 0.
I20260924 03:20:18.987641 2778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e51ed93e024b6ca3f3e6d055a5df03" candidate_uuid: "96fd22ad7a854cc2926b761f91883c6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b079cf51b2b41b3855132ed00448fd2" is_pre_election: true
I20260924 03:20:18.988263 2778 raft_consensus.cc:2454] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fd22ad7a854cc2926b761f91883c6f in term 0.
I20260924 03:20:18.988493 2844 leader_election.cc:302] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3440b59e10804420bc376a68cf05e37f, 96fd22ad7a854cc2926b761f91883c6f; no voters:
I20260924 03:20:18.988929 3110 raft_consensus.cc:2781] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:20:18.989087 3110 raft_consensus.cc:482] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:20:18.989198 3110 raft_consensus.cc:3037] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:18.991550 3110 raft_consensus.cc:504] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:18.992065 3110 leader_election.cc:288] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [CANDIDATE]: Term 1 election: Requested vote from peers 3440b59e10804420bc376a68cf05e37f (127.14.227.131:37117), 8b079cf51b2b41b3855132ed00448fd2 (127.14.227.129:39741)
I20260924 03:20:18.992594 3039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e51ed93e024b6ca3f3e6d055a5df03" candidate_uuid: "96fd22ad7a854cc2926b761f91883c6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3440b59e10804420bc376a68cf05e37f"
I20260924 03:20:18.992616 2778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e51ed93e024b6ca3f3e6d055a5df03" candidate_uuid: "96fd22ad7a854cc2926b761f91883c6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b079cf51b2b41b3855132ed00448fd2"
I20260924 03:20:18.992810 2778 raft_consensus.cc:3037] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:18.992810 3039 raft_consensus.cc:3037] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:18.996163 2778 raft_consensus.cc:2454] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fd22ad7a854cc2926b761f91883c6f in term 1.
I20260924 03:20:18.996163 3039 raft_consensus.cc:2454] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fd22ad7a854cc2926b761f91883c6f in term 1.
I20260924 03:20:18.996564 2844 leader_election.cc:302] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3440b59e10804420bc376a68cf05e37f, 96fd22ad7a854cc2926b761f91883c6f; no voters:
I20260924 03:20:18.996830 3110 raft_consensus.cc:2781] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:18.997344 3110 raft_consensus.cc:686] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 1 LEADER]: Becoming Leader. State: Replica: 96fd22ad7a854cc2926b761f91883c6f, State: Running, Role: LEADER
I20260924 03:20:18.997813 3110 consensus_queue.cc:237] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } }
I20260924 03:20:19.002918 2635 catalog_manager.cc:5495] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fd22ad7a854cc2926b761f91883c6f (127.14.227.130). New cstate: current_term: 1 leader_uuid: "96fd22ad7a854cc2926b761f91883c6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fd22ad7a854cc2926b761f91883c6f" member_type: VOTER last_known_addr { host: "127.14.227.130" port: 46745 } health_report { overall_health: HEALTHY } } }
W20260924 03:20:19.018049 2824 tablet.cc:2369] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:19.030035 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:19.024926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.14.227.130:46745)
0924 03:20:19.025153 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:19.025159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:19.025175 (+ 16us) server_negotiation.cc:407] Connection header received
0924 03:20:19.025271 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:19.025275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:19.025358 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:19.025508 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:19.025864 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:19.026603 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:19.027954 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:19.028355 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:19.028486 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:19.028932 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:19.029419 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:19.029576 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:19.029634 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 03:20:19.029710 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:20:19.037696 2778 raft_consensus.cc:1261] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 1 FOLLOWER]: Refusing update from remote peer 96fd22ad7a854cc2926b761f91883c6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:19.037768 3039 raft_consensus.cc:1261] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 1 FOLLOWER]: Refusing update from remote peer 96fd22ad7a854cc2926b761f91883c6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:19.038753 3113 consensus_queue.cc:1035] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3440b59e10804420bc376a68cf05e37f" member_type: VOTER last_known_addr { host: "127.14.227.131" port: 37117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:19.039207 3110 consensus_queue.cc:1035] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b079cf51b2b41b3855132ed00448fd2" member_type: VOTER last_known_addr { host: "127.14.227.129" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:19.050559 3116 mvcc.cc:205] Tried to move back new op lower bound from 7332741197968445440 to 7332741197819359232. Current Snapshot: MvccSnapshot[applied={T|T < 7332741197968445440}]
I20260924 03:20:19.069895 2635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:19.070225 2635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:20:19.073887 2635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6beda5d89ce464aae1ca934f83215eb: Sending DeleteTablet for 3 replicas of tablet c7e51ed93e024b6ca3f3e6d055a5df03
I20260924 03:20:19.075150 3019 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e51ed93e024b6ca3f3e6d055a5df03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:19 UTC) from {username='slave'} at 127.0.0.1:33808
I20260924 03:20:19.075212 2758 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e51ed93e024b6ca3f3e6d055a5df03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:19 UTC) from {username='slave'} at 127.0.0.1:56000
I20260924 03:20:19.075322 2889 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e51ed93e024b6ca3f3e6d055a5df03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:20:19 UTC) from {username='slave'} at 127.0.0.1:38690
I20260924 03:20:19.076933 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 2696
I20260924 03:20:19.087663 3128 tablet_replica.cc:330] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: stopping tablet replica
I20260924 03:20:19.088214 3128 raft_consensus.cc:2229] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:20:19.089056 3128 raft_consensus.cc:2258] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:19.090219 3127 tablet_replica.cc:330] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: stopping tablet replica
I20260924 03:20:19.090576 3127 raft_consensus.cc:2229] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:19.091001 3127 raft_consensus.cc:2258] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:19.092002 3128 ts_tablet_manager.cc:1905] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:19.092571 3127 ts_tablet_manager.cc:1905] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:19.098245 3127 ts_tablet_manager.cc:1918] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:19.098404 3127 log.cc:1196] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-0/wal/wals/c7e51ed93e024b6ca3f3e6d055a5df03
I20260924 03:20:19.098977 3127 ts_tablet_manager.cc:1939] T c7e51ed93e024b6ca3f3e6d055a5df03 P 8b079cf51b2b41b3855132ed00448fd2: Deleting consensus metadata
I20260924 03:20:19.102146 2623 catalog_manager.cc:4841] TS 8b079cf51b2b41b3855132ed00448fd2 (127.14.227.129:39741): tablet c7e51ed93e024b6ca3f3e6d055a5df03 (table test-table [id=6dbe712c62144ab2b98b95fd17b7174e]) successfully deleted
I20260924 03:20:19.105563 3126 tablet_replica.cc:330] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: stopping tablet replica
I20260924 03:20:19.106072 3126 raft_consensus.cc:2229] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:20:19.106587 3126 raft_consensus.cc:2258] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:20:19.108515 3128 ts_tablet_manager.cc:1918] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:19.108706 3128 log.cc:1196] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-1/wal/wals/c7e51ed93e024b6ca3f3e6d055a5df03
I20260924 03:20:19.109207 3128 ts_tablet_manager.cc:1939] T c7e51ed93e024b6ca3f3e6d055a5df03 P 96fd22ad7a854cc2926b761f91883c6f: Deleting consensus metadata
I20260924 03:20:19.109187 3126 ts_tablet_manager.cc:1905] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:20:19.111121 2623 catalog_manager.cc:4841] TS 96fd22ad7a854cc2926b761f91883c6f (127.14.227.130:46745): tablet c7e51ed93e024b6ca3f3e6d055a5df03 (table test-table [id=6dbe712c62144ab2b98b95fd17b7174e]) successfully deleted
I20260924 03:20:19.113309 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:19.078995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.14.227.129:39741)
0924 03:20:19.079797 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:19.079807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:19.098382 (+ 18575us) server_negotiation.cc:407] Connection header received
0924 03:20:19.098833 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:19.098845 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:19.098992 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:19.099111 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:19.100206 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:19.101743 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:19.105876 (+ 4133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:19.106544 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:19.106669 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:19.111837 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:19.111857 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:19.111872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:19.112416 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:19.112462 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:19.112482 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:19.112647 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:19.112778 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:20:19.112890 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607}
I20260924 03:20:19.114166 2818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:19.116214 3126 ts_tablet_manager.cc:1918] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:20:19.116369 3126 log.cc:1196] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Deleting WAL directory at /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790219707325830-15246-0/minicluster-data/ts-2/wal/wals/c7e51ed93e024b6ca3f3e6d055a5df03
I20260924 03:20:19.116856 3126 ts_tablet_manager.cc:1939] T c7e51ed93e024b6ca3f3e6d055a5df03 P 3440b59e10804420bc376a68cf05e37f: Deleting consensus metadata
I20260924 03:20:19.118608 2623 catalog_manager.cc:4841] TS 3440b59e10804420bc376a68cf05e37f (127.14.227.131:37117): tablet c7e51ed93e024b6ca3f3e6d055a5df03 (table test-table [id=6dbe712c62144ab2b98b95fd17b7174e]) successfully deleted
I20260924 03:20:19.214311 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 2696
I20260924 03:20:19.239389 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 2827
I20260924 03:20:19.248770 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:19.240238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.14.227.130:46745)
0924 03:20:19.240879 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:19.240885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:19.240909 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:20:19.240976 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:19.240982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:19.241076 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:19.241194 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:19.241688 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:19.242930 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:19.244781 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:19.245542 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:19.245671 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:19.247235 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:19.247249 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:19.247263 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:19.247908 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:19.247968 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:19.247990 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:19.248097 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:19.248253 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 03:20:19.248348 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482}
I20260924 03:20:19.249500 2949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:19.362002 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 2827
I20260924 03:20:19.380479 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 2957
I20260924 03:20:19.387583 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:19.381212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.14.227.131:37117)
0924 03:20:19.381396 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:19.381400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:19.381418 (+ 18us) server_negotiation.cc:407] Connection header received
0924 03:20:19.381554 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:19.381557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:19.381640 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:19.381722 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:19.382147 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:19.382850 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:19.384129 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:19.384558 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:19.384663 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:19.385581 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:19.385590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:19.385604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:19.386504 (+ 900us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:19.386602 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:19.386651 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:19.386792 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:19.386996 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 03:20:19.387134 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:20:19.388314 3079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:19.477805 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 2957
I20260924 03:20:19.493851 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 2606
I20260924 03:20:19.495020 2667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:19.602912 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 2606
2026-09-24T03:20:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2855 ms)
[----------] 6 tests from EncryptionPolicyTest (21809 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:19 dist-test-slave-vd0p krb5kdc[3136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:19 dist-test-slave-vd0p krb5kdc[3136](info): set up 2 sockets
Sep 24 03:20:19 dist-test-slave-vd0p krb5kdc[3136](info): commencing operation
krb5kdc: starting...
W20260924 03:20:20.698527 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:20 dist-test-slave-vd0p krb5kdc[3136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:20.951927 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:21.300071 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:21.300523 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:21.300730 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:21.308562 3149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:21.308727 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:21.308823 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:21.308955 3149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:21.309054 3149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:21.319960 3149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:21.321332 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:21.322798 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:21.335731 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:21.335763 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:21.335702 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:21.336745 3149 server_base.cc:1034] running on GCE node
I20260924 03:20:21.337615 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:21.339138 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:21.340351 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790220021340307 us; error 64 us; skew 500 ppm
I20260924 03:20:21.340713 3149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:21 dist-test-slave-vd0p krb5kdc[3136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:21.344072 3149 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:21.345629 3149 webserver.cc:460] Webserver started at http://127.0.0.1:45449/ using document root <none> and password file <none>
I20260924 03:20:21.346431 3149 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:21.346580 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:21.346909 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:21.350010 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "338c5cadc1ea41f1b8822a7d47484442"
format_stamp: "Formatted at 2026-09-24 03:20:21 on dist-test-slave-vd0p"
server_key: "9cbb5cd662d62622303ef453654f3c8e"
server_key_iv: "c3eab3df1e68b5bd461c623bedadd3a7"
server_key_version: "encryptionkey@0"
I20260924 03:20:21.350832 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "338c5cadc1ea41f1b8822a7d47484442"
format_stamp: "Formatted at 2026-09-24 03:20:21 on dist-test-slave-vd0p"
server_key: "9cbb5cd662d62622303ef453654f3c8e"
server_key_iv: "c3eab3df1e68b5bd461c623bedadd3a7"
server_key_version: "encryptionkey@0"
I20260924 03:20:21.356130 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:20:21.359939 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:21.361528 3149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:20:21.361735 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "338c5cadc1ea41f1b8822a7d47484442"
format_stamp: "Formatted at 2026-09-24 03:20:21 on dist-test-slave-vd0p"
server_key: "9cbb5cd662d62622303ef453654f3c8e"
server_key_iv: "c3eab3df1e68b5bd461c623bedadd3a7"
server_key_version: "encryptionkey@0"
I20260924 03:20:21.361955 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:21.383725 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:21.389780 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:21.390218 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:21.410550 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44327
I20260924 03:20:21.410594 3215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44327 every 8 connection(s)
I20260924 03:20:21.412081 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:21.416370 3216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:21.422185 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 3149
I20260924 03:20:21.422399 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:21.423153 15246 external_mini_cluster.cc:1428] Setting key b69176fc48fc0c081a14de794f6516a4
I20260924 03:20:21.432107 3216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442: Bootstrap starting.
I20260924 03:20:21.436010 3216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:21.437404 3216 log.cc:824] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:20:21 dist-test-slave-vd0p krb5kdc[3136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:20:21.440770 3216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442: No bootstrap required, opened a new log
I20260924 03:20:21.447690 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:21.425282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38749 (local address 127.0.0.1:44327)
0924 03:20:21.426054 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:21.426069 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:21.426111 (+ 42us) server_negotiation.cc:407] Connection header received
0924 03:20:21.427812 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:21.427834 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:21.428343 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:21.428681 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:21.430529 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:21.431835 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:21.433555 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:21.434318 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:21.434456 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:21.439817 (+ 5361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:21.439825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:21.439839 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:21.439876 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:21.443080 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:21.444076 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:21.444086 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:21.444092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:21.444258 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:21.444807 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:21.444811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:21.444813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:21.445358 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:21.445660 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:21.446229 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 03:20:21.446647 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":304,"threads_started":1}
I20260924 03:20:21.450371 3216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338c5cadc1ea41f1b8822a7d47484442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44327 } }
I20260924 03:20:21.450857 3216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:21.450965 3216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338c5cadc1ea41f1b8822a7d47484442, State: Initialized, Role: FOLLOWER
I20260924 03:20:21.451650 3216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338c5cadc1ea41f1b8822a7d47484442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44327 } }
I20260924 03:20:21.451957 3216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:21.452129 3216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:21.452317 3216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:21.452348 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:21.454650 3216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338c5cadc1ea41f1b8822a7d47484442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44327 } }
I20260924 03:20:21.455166 3216 leader_election.cc:302] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 338c5cadc1ea41f1b8822a7d47484442; no voters:
I20260924 03:20:21.455917 3216 leader_election.cc:288] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:21.456022 3224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:21.457288 3224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [term 1 LEADER]: Becoming Leader. State: Replica: 338c5cadc1ea41f1b8822a7d47484442, State: Running, Role: LEADER
I20260924 03:20:21.457917 3224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338c5cadc1ea41f1b8822a7d47484442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44327 } }
I20260924 03:20:21.458422 3216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:21.461068 3226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "338c5cadc1ea41f1b8822a7d47484442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338c5cadc1ea41f1b8822a7d47484442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44327 } } }
I20260924 03:20:21.461552 3226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:21.464362 3227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 338c5cadc1ea41f1b8822a7d47484442. Latest consensus state: current_term: 1 leader_uuid: "338c5cadc1ea41f1b8822a7d47484442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338c5cadc1ea41f1b8822a7d47484442" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44327 } } }
I20260924 03:20:21.464686 3227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:21.465648 3231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:21.476549 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:21.457969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54503 (local address 127.0.0.1:44327)
0924 03:20:21.458792 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:21.458801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:21.458828 (+ 27us) server_negotiation.cc:407] Connection header received
0924 03:20:21.459058 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:21.459063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:21.459148 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:21.459264 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:21.461330 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:21.462661 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:21.464984 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:21.465517 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:21.465638 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:21.469691 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:21.469700 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:21.469707 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:21.469757 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:21.473514 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:21.474427 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:21.474432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:21.474436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:21.474554 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:21.475031 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:21.475036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:21.475039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:21.475346 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:21.475608 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:21.475981 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 03:20:21.476143 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706}
I20260924 03:20:21.484426 3231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:21.494227 3231 catalog_manager.cc:1347] Generated new cluster ID: 9bc8b97073fc48c891579c9630f793b1
I20260924 03:20:21.494347 3231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:21.526717 3231 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:21.528069 3231 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:21.539520 3231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 338c5cadc1ea41f1b8822a7d47484442: Generated new TSK 0
I20260924 03:20:21.540371 3231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:21.550158 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3149
I20260924 03:20:21.551407 3211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:21.646090 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 3149
W20260924 03:20:21.663039 3134 connection.cc:535] client connection to 127.0.0.1:44327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44327): Connection reset by peer (error 104)
2026-09-24T03:20:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:21 dist-test-slave-vd0p krb5kdc[3249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:21 dist-test-slave-vd0p krb5kdc[3249](info): set up 2 sockets
Sep 24 03:20:21 dist-test-slave-vd0p krb5kdc[3249](info): commencing operation
krb5kdc: starting...
W20260924 03:20:22.734395 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:22 dist-test-slave-vd0p krb5kdc[3249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:22.973892 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:23.303257 3262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:23.303572 3262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:23.303790 3262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:23.311163 3262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:23.311276 3262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:23.311384 3262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:23.311496 3262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:23.311589 3262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:23.322748 3262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:23.324075 3262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:23.325390 3262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:23.337527 3267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:23.337682 3268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:23.337695 3270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:23.338554 3262 server_base.cc:1034] running on GCE node
I20260924 03:20:23.339375 3262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:23.340751 3262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:23.342034 3262 hybrid_clock.cc:648] HybridClock initialized: now 1790220023341979 us; error 54 us; skew 500 ppm
I20260924 03:20:23.342440 3262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:23 dist-test-slave-vd0p krb5kdc[3249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:23.345691 3262 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:23.347276 3262 webserver.cc:460] Webserver started at http://127.0.0.1:38101/ using document root <none> and password file <none>
I20260924 03:20:23.347949 3262 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:23.348116 3262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:23.348434 3262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:23.351512 3262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "c3a73269152d462798c960661b926956"
format_stamp: "Formatted at 2026-09-24 03:20:23 on dist-test-slave-vd0p"
server_key: "b430712e241cc573dc35e8ac26b37cee"
server_key_iv: "8639ef53db854b757d98175d424a4ef0"
server_key_version: "encryptionkey@0"
I20260924 03:20:23.352344 3262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "c3a73269152d462798c960661b926956"
format_stamp: "Formatted at 2026-09-24 03:20:23 on dist-test-slave-vd0p"
server_key: "b430712e241cc573dc35e8ac26b37cee"
server_key_iv: "8639ef53db854b757d98175d424a4ef0"
server_key_version: "encryptionkey@0"
I20260924 03:20:23.357458 3262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:20:23.360772 3277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:23.362109 3262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 03:20:23.362324 3262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "c3a73269152d462798c960661b926956"
format_stamp: "Formatted at 2026-09-24 03:20:23 on dist-test-slave-vd0p"
server_key: "b430712e241cc573dc35e8ac26b37cee"
server_key_iv: "8639ef53db854b757d98175d424a4ef0"
server_key_version: "encryptionkey@0"
I20260924 03:20:23.362510 3262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:23.380540 3262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:23.384383 3262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:23.384668 3262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:23.406082 3262 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45227
I20260924 03:20:23.406096 3328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45227 every 8 connection(s)
I20260924 03:20:23.407517 3262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:23.411098 3329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:23.417282 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 3262
I20260924 03:20:23.417455 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:23.417876 15246 external_mini_cluster.cc:1428] Setting key 9e1a5b040e36ef59f61fc2860c9956c4
I20260924 03:20:23.421506 3329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956: Bootstrap starting.
I20260924 03:20:23.424688 3329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:23.425930 3329 log.cc:824] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:23.429073 3329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956: No bootstrap required, opened a new log
I20260924 03:20:23.431452 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:23.419617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60595 (local address 127.0.0.1:45227)
0924 03:20:23.420377 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:23.420388 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:23.420427 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:20:23.422149 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:23.422170 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:23.422537 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:23.422841 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:23.423458 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:23.424701 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:23.426692 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:23.427563 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:23.427714 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:23.429166 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:23.429178 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:23.429195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:23.429733 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:23.429801 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:23.429824 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:23.429933 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:23.430281 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 03:20:23.430456 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":357,"threads_started":1}
I20260924 03:20:23.435348 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:23.437216 3329 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a73269152d462798c960661b926956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45227 } }
I20260924 03:20:23.437566 3329 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:23.437681 3329 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a73269152d462798c960661b926956, State: Initialized, Role: FOLLOWER
I20260924 03:20:23.438357 3329 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a73269152d462798c960661b926956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45227 } }
I20260924 03:20:23.438602 3329 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:23.438781 3329 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:23.438985 3329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:23.441928 3329 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a73269152d462798c960661b926956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45227 } }
I20260924 03:20:23.442529 3329 leader_election.cc:302] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3a73269152d462798c960661b926956; no voters:
I20260924 03:20:23.443197 3329 leader_election.cc:288] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:23.443297 3339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:23.444506 3339 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [term 1 LEADER]: Becoming Leader. State: Replica: c3a73269152d462798c960661b926956, State: Running, Role: LEADER
I20260924 03:20:23.445119 3339 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a73269152d462798c960661b926956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45227 } }
I20260924 03:20:23.445542 3329 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:23.447127 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:23.439712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36705 (local address 127.0.0.1:45227)
0924 03:20:23.439883 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:23.439889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:23.440376 (+ 487us) server_negotiation.cc:407] Connection header received
0924 03:20:23.440487 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:23.440492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:23.440562 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:23.440676 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:23.441102 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:23.442197 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:23.443691 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:23.444378 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:23.444494 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:23.446163 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:23.446170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:23.446176 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:23.446205 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:23.446268 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:23.446287 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:23.446438 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:23.446689 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 03:20:23.446750 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 03:20:23.448854 3341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3a73269152d462798c960661b926956. Latest consensus state: current_term: 1 leader_uuid: "c3a73269152d462798c960661b926956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a73269152d462798c960661b926956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45227 } } }
I20260924 03:20:23.449339 3340 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3a73269152d462798c960661b926956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a73269152d462798c960661b926956" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45227 } } }
I20260924 03:20:23.449688 3340 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:23.449355 3341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:23.454583 3345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:23.459818 3345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:23.469507 3345 catalog_manager.cc:1347] Generated new cluster ID: 7d0bf367a46848c7973b673e9db3ccad
I20260924 03:20:23.469672 3345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:23.481627 3345 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:23.483528 3345 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:23.490981 3345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3a73269152d462798c960661b926956: Generated new TSK 0
I20260924 03:20:23.492012 3345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:23.508610 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3262
W20260924 03:20:23.509800 3324 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60595
W20260924 03:20:23.510638 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3262: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:23.510823 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 3262
W20260924 03:20:23.527383 3247 connection.cc:535] client connection to 127.0.0.1:45227 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45227): Connection reset by peer (error 104)
2026-09-24T03:20:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1864 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:23 dist-test-slave-vd0p krb5kdc[3361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:23 dist-test-slave-vd0p krb5kdc[3361](info): set up 2 sockets
Sep 24 03:20:23 dist-test-slave-vd0p krb5kdc[3361](info): commencing operation
krb5kdc: starting...
W20260924 03:20:24.598330 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:24 dist-test-slave-vd0p krb5kdc[3361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:24.838243 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:25.159437 3374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:25.159793 3374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:25.159968 3374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:25.166971 3374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:25.167120 3374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:25.167232 3374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:25.167346 3374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:25.167434 3374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:25.176934 3374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:25.178228 3374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:25.179524 3374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:25.190953 3379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:25.190914 3380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:25.191054 3382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:25.192014 3374 server_base.cc:1034] running on GCE node
I20260924 03:20:25.192816 3374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:25.194125 3374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:25.195322 3374 hybrid_clock.cc:648] HybridClock initialized: now 1790220025195283 us; error 32 us; skew 500 ppm
I20260924 03:20:25.195669 3374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:25 dist-test-slave-vd0p krb5kdc[3361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:25.203027 3374 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:25.204520 3374 webserver.cc:460] Webserver started at http://127.0.0.1:43189/ using document root <none> and password file <none>
I20260924 03:20:25.205173 3374 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:25.205312 3374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:25.205621 3374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:25.208547 3374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "01202a0fab8c41868a44b52ff9e2ed8a"
format_stamp: "Formatted at 2026-09-24 03:20:25 on dist-test-slave-vd0p"
server_key: "98d83112a97ec42eeda56827b5da5f4d"
server_key_iv: "f8407ec0f84ee68000c2f2560aa5418d"
server_key_version: "encryptionkey@0"
I20260924 03:20:25.209326 3374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "01202a0fab8c41868a44b52ff9e2ed8a"
format_stamp: "Formatted at 2026-09-24 03:20:25 on dist-test-slave-vd0p"
server_key: "98d83112a97ec42eeda56827b5da5f4d"
server_key_iv: "f8407ec0f84ee68000c2f2560aa5418d"
server_key_version: "encryptionkey@0"
I20260924 03:20:25.214460 3374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 03:20:25.217779 3389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:25.219125 3374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:25.219310 3374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "01202a0fab8c41868a44b52ff9e2ed8a"
format_stamp: "Formatted at 2026-09-24 03:20:25 on dist-test-slave-vd0p"
server_key: "98d83112a97ec42eeda56827b5da5f4d"
server_key_iv: "f8407ec0f84ee68000c2f2560aa5418d"
server_key_version: "encryptionkey@0"
I20260924 03:20:25.219527 3374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:25.225183 3374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:25.229421 3374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:25.229717 3374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:25.247843 3374 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46737
I20260924 03:20:25.247890 3440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46737 every 8 connection(s)
I20260924 03:20:25.249228 3374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:25.250953 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 3374
I20260924 03:20:25.251127 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:25.251659 15246 external_mini_cluster.cc:1428] Setting key b2f21b388354ee04c78f420d9ff07567
I20260924 03:20:25.253616 3441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:25.259209 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:25.253462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58325 (local address 127.0.0.1:46737)
0924 03:20:25.254277 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:25.254289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:25.254326 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:20:25.255810 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:25.255829 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:25.256152 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:25.256562 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:25.257602 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:25.257608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:25.257620 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:25.257634 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:25.257707 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:25.257726 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:25.257983 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:25.258126 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 03:20:25.258299 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":348,"threads_started":1}
I20260924 03:20:25.262832 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:25.264962 3441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a: Bootstrap starting.
I20260924 03:20:25.268277 3441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:25.269546 3441 log.cc:824] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:25.269739 3444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:25.267098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52719 (local address 127.0.0.1:46737)
0924 03:20:25.267296 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:25.267304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:25.267804 (+ 500us) server_negotiation.cc:407] Connection header received
0924 03:20:25.267985 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:25.267988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:25.268040 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:25.268109 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:25.269023 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:25.269027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:25.269030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:25.269043 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:25.269078 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:25.269092 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:25.269172 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:25.269312 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 03:20:25.269357 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 03:20:25.272113 3441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a: No bootstrap required, opened a new log
I20260924 03:20:25.278851 3441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01202a0fab8c41868a44b52ff9e2ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46737 } }
I20260924 03:20:25.279182 3441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:25.279292 3441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01202a0fab8c41868a44b52ff9e2ed8a, State: Initialized, Role: FOLLOWER
I20260924 03:20:25.279825 3441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01202a0fab8c41868a44b52ff9e2ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46737 } }
I20260924 03:20:25.280051 3441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:25.280195 3441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:25.280326 3441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:25.282667 3441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01202a0fab8c41868a44b52ff9e2ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46737 } }
I20260924 03:20:25.283097 3441 leader_election.cc:302] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01202a0fab8c41868a44b52ff9e2ed8a; no voters:
I20260924 03:20:25.283618 3441 leader_election.cc:288] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:25.283742 3451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:25.284395 3451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [term 1 LEADER]: Becoming Leader. State: Replica: 01202a0fab8c41868a44b52ff9e2ed8a, State: Running, Role: LEADER
I20260924 03:20:25.284926 3451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01202a0fab8c41868a44b52ff9e2ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46737 } }
I20260924 03:20:25.285507 3441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:25.287811 3453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01202a0fab8c41868a44b52ff9e2ed8a. Latest consensus state: current_term: 1 leader_uuid: "01202a0fab8c41868a44b52ff9e2ed8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01202a0fab8c41868a44b52ff9e2ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46737 } } }
I20260924 03:20:25.288287 3453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:25.289186 3452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01202a0fab8c41868a44b52ff9e2ed8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01202a0fab8c41868a44b52ff9e2ed8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46737 } } }
I20260924 03:20:25.289675 3452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:25.290786 3459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:25.296029 3459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:25.304567 3459 catalog_manager.cc:1347] Generated new cluster ID: 38bf5fbea10f4148bc003219d55f7805
I20260924 03:20:25.304692 3459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:25.325121 3459 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:25.325671 3459 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:25.334867 3459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01202a0fab8c41868a44b52ff9e2ed8a: Generated new TSK 0
I20260924 03:20:25.335737 3459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:25.341003 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3374
W20260924 03:20:25.342000 3436 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58325
W20260924 03:20:25.343062 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3374: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:25.343245 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 3374
2026-09-24T03:20:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:25 dist-test-slave-vd0p krb5kdc[3472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:25 dist-test-slave-vd0p krb5kdc[3472](info): set up 2 sockets
Sep 24 03:20:25 dist-test-slave-vd0p krb5kdc[3472](info): commencing operation
krb5kdc: starting...
W20260924 03:20:26.438334 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:26 dist-test-slave-vd0p krb5kdc[3472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:26.687592 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:27.011293 3485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:27.011684 3485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:27.011849 3485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:27.019375 3485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:27.019541 3485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:27.019655 3485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:27.019795 3485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:27.019881 3485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:27.029893 3485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:27.031306 3485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:27.032667 3485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:27.044724 3493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:27.044757 3491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:27.044981 3490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:27.045645 3485 server_base.cc:1034] running on GCE node
I20260924 03:20:27.046474 3485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:27.047885 3485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:27.049121 3485 hybrid_clock.cc:648] HybridClock initialized: now 1790220027049075 us; error 61 us; skew 500 ppm
I20260924 03:20:27.049471 3485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:27 dist-test-slave-vd0p krb5kdc[3472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:27.052704 3485 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:27.054248 3485 webserver.cc:460] Webserver started at http://127.0.0.1:41501/ using document root <none> and password file <none>
I20260924 03:20:27.054919 3485 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:27.055050 3485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:27.055353 3485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:27.058329 3485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "bb5a9994d69c4f37a96bf2eac8350fe9"
format_stamp: "Formatted at 2026-09-24 03:20:27 on dist-test-slave-vd0p"
server_key: "327fe3fe4358d71ecc8df531420f2c7f"
server_key_iv: "dc24080f98c881a49fec8ab1d3c62ebd"
server_key_version: "encryptionkey@0"
I20260924 03:20:27.059093 3485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "bb5a9994d69c4f37a96bf2eac8350fe9"
format_stamp: "Formatted at 2026-09-24 03:20:27 on dist-test-slave-vd0p"
server_key: "327fe3fe4358d71ecc8df531420f2c7f"
server_key_iv: "dc24080f98c881a49fec8ab1d3c62ebd"
server_key_version: "encryptionkey@0"
I20260924 03:20:27.064200 3485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:20:27.067674 3500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:27.068980 3485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:20:27.069155 3485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "bb5a9994d69c4f37a96bf2eac8350fe9"
format_stamp: "Formatted at 2026-09-24 03:20:27 on dist-test-slave-vd0p"
server_key: "327fe3fe4358d71ecc8df531420f2c7f"
server_key_iv: "dc24080f98c881a49fec8ab1d3c62ebd"
server_key_version: "encryptionkey@0"
I20260924 03:20:27.069351 3485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:27.087852 3485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:27.091921 3485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:27.092211 3485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:27.110203 3485 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36173
I20260924 03:20:27.110252 3551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36173 every 8 connection(s)
I20260924 03:20:27.111567 3485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:27.112094 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 3485
I20260924 03:20:27.112322 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:27.112913 15246 external_mini_cluster.cc:1428] Setting key 1855c9d46972fd34e6a7df1b68250655
I20260924 03:20:27.116266 3552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:20:27 dist-test-slave-vd0p krb5kdc[3472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:20:27.126852 3552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9: Bootstrap starting.
I20260924 03:20:27.130162 3552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:27.131428 3552 log.cc:824] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:27.134845 3552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9: No bootstrap required, opened a new log
I20260924 03:20:27.135161 3555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:27.114740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38265 (local address 127.0.0.1:36173)
0924 03:20:27.115479 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:27.115492 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:27.115532 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:20:27.117190 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:27.117213 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:27.117599 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:27.117997 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:27.119461 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:27.120812 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:27.122126 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:27.122811 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:27.123014 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:27.127406 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:27.127414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:27.127428 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:27.127463 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:27.130648 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:27.131576 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:27.131587 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:27.131597 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:27.131770 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:27.132235 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:27.132242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:27.132244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:27.133017 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:27.133231 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:27.133679 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 03:20:27.134086 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":533,"thread_start_us":315,"threads_started":1}
I20260924 03:20:27.138800 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:27.144668 3552 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5a9994d69c4f37a96bf2eac8350fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } }
I20260924 03:20:27.145216 3552 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:27.145321 3552 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5a9994d69c4f37a96bf2eac8350fe9, State: Initialized, Role: FOLLOWER
I20260924 03:20:27.145913 3552 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5a9994d69c4f37a96bf2eac8350fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } }
I20260924 03:20:27.146210 3552 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:27.146365 3552 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:27.146536 3552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:27.149382 3552 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5a9994d69c4f37a96bf2eac8350fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } }
I20260924 03:20:27.150025 3552 leader_election.cc:302] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb5a9994d69c4f37a96bf2eac8350fe9; no voters:
I20260924 03:20:27.150723 3552 leader_election.cc:288] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:27.150813 3562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:27.152153 3562 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [term 1 LEADER]: Becoming Leader. State: Replica: bb5a9994d69c4f37a96bf2eac8350fe9, State: Running, Role: LEADER
I20260924 03:20:27.152849 3562 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5a9994d69c4f37a96bf2eac8350fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } }
I20260924 03:20:27.153252 3552 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:27.155858 3564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb5a9994d69c4f37a96bf2eac8350fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5a9994d69c4f37a96bf2eac8350fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } } }
I20260924 03:20:27.155958 3563 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5a9994d69c4f37a96bf2eac8350fe9. Latest consensus state: current_term: 1 leader_uuid: "bb5a9994d69c4f37a96bf2eac8350fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5a9994d69c4f37a96bf2eac8350fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } } }
I20260924 03:20:27.156243 3564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:27.156342 3563 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:27.157253 3567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:27.161012 3555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:27.144100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36011 (local address 127.0.0.1:36173)
0924 03:20:27.144291 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:27.144296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:27.144730 (+ 434us) server_negotiation.cc:407] Connection header received
0924 03:20:27.144864 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:27.144868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:27.144946 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:27.145070 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:27.146882 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:27.148026 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:27.149209 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:27.149889 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:27.150065 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:27.153013 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:27.153024 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:27.153028 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:27.153080 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:27.157074 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:27.157978 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:27.157986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:27.157991 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:27.158195 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:27.158679 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:27.158684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:27.158687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:27.159045 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:27.159373 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:27.160123 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 03:20:27.160439 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:20:27.171845 3567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:27.182724 3567 catalog_manager.cc:1347] Generated new cluster ID: b515a3e7af38439f9399cb3d44b6e10c
I20260924 03:20:27.182858 3567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:27.196656 3567 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:27.198550 3567 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:27.207146 3567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb5a9994d69c4f37a96bf2eac8350fe9: Generated new TSK 0
I20260924 03:20:27.208256 3567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:27.221025 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3485
I20260924 03:20:27.222158 3547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:27.313853 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 3485
2026-09-24T03:20:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:27 dist-test-slave-vd0p krb5kdc[3584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:27 dist-test-slave-vd0p krb5kdc[3584](info): set up 2 sockets
Sep 24 03:20:27 dist-test-slave-vd0p krb5kdc[3584](info): commencing operation
krb5kdc: starting...
W20260924 03:20:28.410276 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:28 dist-test-slave-vd0p krb5kdc[3584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:28.663203 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:28.983072 3597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:28.983407 3597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:28.983578 3597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:28.991106 3597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:28.991243 3597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:28.991359 3597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:28.991447 3597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:28.991515 3597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:29.000939 3597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:29.002170 3597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:29.003471 3597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:29.015017 3602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:29.015074 3603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:29.015105 3605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:29.015966 3597 server_base.cc:1034] running on GCE node
I20260924 03:20:29.016620 3597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:29.017940 3597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:29.019194 3597 hybrid_clock.cc:648] HybridClock initialized: now 1790220029019153 us; error 33 us; skew 500 ppm
I20260924 03:20:29.019526 3597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:29 dist-test-slave-vd0p krb5kdc[3584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:29.022606 3597 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:29.024084 3597 webserver.cc:460] Webserver started at http://127.0.0.1:45027/ using document root <none> and password file <none>
I20260924 03:20:29.024725 3597 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:29.024873 3597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:29.025180 3597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:29.028169 3597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "a68851c814864245b9e47980e3012d29"
format_stamp: "Formatted at 2026-09-24 03:20:29 on dist-test-slave-vd0p"
server_key: "44fd368932e0640f31a398ae5b8b9be2"
server_key_iv: "c8d153656e859c0b510c12357c8f0ac9"
server_key_version: "encryptionkey@0"
I20260924 03:20:29.028941 3597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "a68851c814864245b9e47980e3012d29"
format_stamp: "Formatted at 2026-09-24 03:20:29 on dist-test-slave-vd0p"
server_key: "44fd368932e0640f31a398ae5b8b9be2"
server_key_iv: "c8d153656e859c0b510c12357c8f0ac9"
server_key_version: "encryptionkey@0"
I20260924 03:20:29.034144 3597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 03:20:29.037606 3612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:29.039027 3597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:29.039207 3597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "a68851c814864245b9e47980e3012d29"
format_stamp: "Formatted at 2026-09-24 03:20:29 on dist-test-slave-vd0p"
server_key: "44fd368932e0640f31a398ae5b8b9be2"
server_key_iv: "c8d153656e859c0b510c12357c8f0ac9"
server_key_version: "encryptionkey@0"
I20260924 03:20:29.039405 3597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:29.062698 3597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:29.067196 3597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:29.067499 3597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:29.091717 3597 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37903
I20260924 03:20:29.091770 3663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37903 every 8 connection(s)
I20260924 03:20:29.093169 3597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:29.095252 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 3597
I20260924 03:20:29.095448 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:29.096015 15246 external_mini_cluster.cc:1428] Setting key 6ed71ca318ca4e251b89b28471a1b1c8
I20260924 03:20:29.097771 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:29.108958 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:29.098031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57039 (local address 127.0.0.1:37903)
0924 03:20:29.098794 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:29.098807 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:29.098847 (+ 40us) server_negotiation.cc:407] Connection header received
0924 03:20:29.100467 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:29.100487 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:29.100887 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:29.101291 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:29.101826 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:29.103217 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:29.104535 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:29.105198 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:29.105405 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:29.106675 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:29.106683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:29.106701 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:29.107269 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:29.107351 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:29.107381 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:29.107538 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:29.107755 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 03:20:29.108020 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":331,"threads_started":1}
I20260924 03:20:29.109227 3664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29: Bootstrap starting.
I20260924 03:20:29.112622 3664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:29.112666 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:29.113628 3664 log.cc:824] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:29.116532 3664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29: No bootstrap required, opened a new log
I20260924 03:20:29.123980 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:29.117023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58811 (local address 127.0.0.1:37903)
0924 03:20:29.117249 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:29.117258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:29.117716 (+ 458us) server_negotiation.cc:407] Connection header received
0924 03:20:29.117915 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:29.117919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:29.118034 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:29.118112 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:29.118736 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:29.119577 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:29.121484 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:29.122014 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:29.122125 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:29.123245 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:29.123249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:29.123252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:29.123274 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:29.123311 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:29.123324 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:29.123402 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:29.123549 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 03:20:29.123627 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:20:29.126892 3664 raft_consensus.cc:348] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68851c814864245b9e47980e3012d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37903 } }
I20260924 03:20:29.127326 3664 raft_consensus.cc:374] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:29.127440 3664 raft_consensus.cc:729] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68851c814864245b9e47980e3012d29, State: Initialized, Role: FOLLOWER
I20260924 03:20:29.128014 3664 consensus_queue.cc:260] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68851c814864245b9e47980e3012d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37903 } }
I20260924 03:20:29.128248 3664 raft_consensus.cc:388] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:29.128412 3664 raft_consensus.cc:482] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:29.128602 3664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:29.130888 3664 raft_consensus.cc:504] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68851c814864245b9e47980e3012d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37903 } }
I20260924 03:20:29.131536 3664 leader_election.cc:302] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a68851c814864245b9e47980e3012d29; no voters:
I20260924 03:20:29.132138 3664 leader_election.cc:288] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:29.132278 3674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:29.133090 3674 raft_consensus.cc:686] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [term 1 LEADER]: Becoming Leader. State: Replica: a68851c814864245b9e47980e3012d29, State: Running, Role: LEADER
I20260924 03:20:29.133598 3674 consensus_queue.cc:237] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68851c814864245b9e47980e3012d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37903 } }
I20260924 03:20:29.134508 3664 sys_catalog.cc:564] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:29.136746 3675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a68851c814864245b9e47980e3012d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68851c814864245b9e47980e3012d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37903 } } }
I20260924 03:20:29.137135 3675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:29.140347 3676 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a68851c814864245b9e47980e3012d29. Latest consensus state: current_term: 1 leader_uuid: "a68851c814864245b9e47980e3012d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68851c814864245b9e47980e3012d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37903 } } }
I20260924 03:20:29.140708 3676 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:29.141428 3683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:29.146685 3683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:29.155820 3683 catalog_manager.cc:1347] Generated new cluster ID: 2cb7f97a2bdd41998063efe0736c71a4
I20260924 03:20:29.155962 3683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:29.173650 3683 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:29.175027 3683 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:29.182950 3683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a68851c814864245b9e47980e3012d29: Generated new TSK 0
I20260924 03:20:29.183784 3683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:29.196833 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3597
W20260924 03:20:29.198076 3659 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57039
W20260924 03:20:29.198820 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3597: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:29.198987 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 3597
2026-09-24T03:20:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:29 dist-test-slave-vd0p krb5kdc[3695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:29 dist-test-slave-vd0p krb5kdc[3695](info): set up 2 sockets
Sep 24 03:20:29 dist-test-slave-vd0p krb5kdc[3695](info): commencing operation
krb5kdc: starting...
W20260924 03:20:30.285938 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:30 dist-test-slave-vd0p krb5kdc[3695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:30.517174 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:30.844488 3708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:30.844887 3708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:30.845065 3708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:30.852476 3708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:30.852662 3708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:30.852773 3708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:30.852882 3708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:30.852967 3708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:30.863128 3708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:30.864506 3708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:30.865866 3708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:30.878242 3713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:30.878216 3716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:30.878137 3714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:30.879071 3708 server_base.cc:1034] running on GCE node
I20260924 03:20:30.879976 3708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:30.881451 3708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:30.882700 3708 hybrid_clock.cc:648] HybridClock initialized: now 1790220030882659 us; error 38 us; skew 500 ppm
I20260924 03:20:30.883077 3708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:30 dist-test-slave-vd0p krb5kdc[3695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:30.894279 3708 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:30.896096 3708 webserver.cc:460] Webserver started at http://127.0.0.1:45265/ using document root <none> and password file <none>
I20260924 03:20:30.896847 3708 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:30.897008 3708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:30.897338 3708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:30.900595 3708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "153d4d559bd844ae86d48b636fa7df6b"
format_stamp: "Formatted at 2026-09-24 03:20:30 on dist-test-slave-vd0p"
server_key: "d977a1122a39df2bbc4c9fd56201192e"
server_key_iv: "0bb2d246177e74518ff929d6e1745f17"
server_key_version: "encryptionkey@0"
I20260924 03:20:30.901386 3708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "153d4d559bd844ae86d48b636fa7df6b"
format_stamp: "Formatted at 2026-09-24 03:20:30 on dist-test-slave-vd0p"
server_key: "d977a1122a39df2bbc4c9fd56201192e"
server_key_iv: "0bb2d246177e74518ff929d6e1745f17"
server_key_version: "encryptionkey@0"
I20260924 03:20:30.906715 3708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:20:30.910287 3723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:30.911662 3708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:20:30.911851 3708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "153d4d559bd844ae86d48b636fa7df6b"
format_stamp: "Formatted at 2026-09-24 03:20:30 on dist-test-slave-vd0p"
server_key: "d977a1122a39df2bbc4c9fd56201192e"
server_key_iv: "0bb2d246177e74518ff929d6e1745f17"
server_key_version: "encryptionkey@0"
I20260924 03:20:30.912053 3708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:30.918030 3708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:30.922025 3708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:30.922319 3708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:30.941083 3708 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34005
I20260924 03:20:30.941174 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34005 every 8 connection(s)
I20260924 03:20:30.942503 3708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:30.946316 3775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:30.951418 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 3708
I20260924 03:20:30.951614 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:30.952068 15246 external_mini_cluster.cc:1428] Setting key f35d8b380013f5019666b5ff482b3304
I20260924 03:20:30.958259 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b: Bootstrap starting.
I20260924 03:20:30.961067 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:30.953932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53533 (local address 127.0.0.1:34005)
0924 03:20:30.954795 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:30.954810 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:30.954858 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:20:30.957205 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:30.957233 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:30.957665 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:30.958139 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:30.959136 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:30.959146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:30.959162 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:30.959186 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:30.959251 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:30.959276 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:30.959482 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:30.959608 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 03:20:30.959785 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":324,"threads_started":1}
I20260924 03:20:30.962574 3775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:30.963757 3775 log.cc:824] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:30.964784 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:30.966917 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b: No bootstrap required, opened a new log
I20260924 03:20:30.972297 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:30.969843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55299 (local address 127.0.0.1:34005)
0924 03:20:30.970058 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:30.970079 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:30.970319 (+ 240us) server_negotiation.cc:407] Connection header received
0924 03:20:30.970521 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:30.970525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:30.970574 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:30.970683 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:30.971621 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:30.971627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:30.971632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:30.971650 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:30.971698 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:30.971716 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:30.971856 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:30.971954 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:20:30.972017 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:20:30.975555 3775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d4d559bd844ae86d48b636fa7df6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34005 } }
I20260924 03:20:30.975919 3775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:30.976017 3775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153d4d559bd844ae86d48b636fa7df6b, State: Initialized, Role: FOLLOWER
I20260924 03:20:30.976698 3775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d4d559bd844ae86d48b636fa7df6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34005 } }
I20260924 03:20:30.976934 3775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:30.977082 3775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:30.977243 3775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:30.979545 3775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d4d559bd844ae86d48b636fa7df6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34005 } }
I20260924 03:20:30.980062 3775 leader_election.cc:302] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 153d4d559bd844ae86d48b636fa7df6b; no voters:
I20260924 03:20:30.980608 3775 leader_election.cc:288] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:30.980770 3785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:30.981427 3785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [term 1 LEADER]: Becoming Leader. State: Replica: 153d4d559bd844ae86d48b636fa7df6b, State: Running, Role: LEADER
I20260924 03:20:30.981923 3785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d4d559bd844ae86d48b636fa7df6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34005 } }
I20260924 03:20:30.982676 3775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:30.985037 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 153d4d559bd844ae86d48b636fa7df6b. Latest consensus state: current_term: 1 leader_uuid: "153d4d559bd844ae86d48b636fa7df6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d4d559bd844ae86d48b636fa7df6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34005 } } }
I20260924 03:20:30.985016 3786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "153d4d559bd844ae86d48b636fa7df6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "153d4d559bd844ae86d48b636fa7df6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34005 } } }
I20260924 03:20:30.985507 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:30.985678 3786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:30.986572 3790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:30.992759 3790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:31.003700 3790 catalog_manager.cc:1347] Generated new cluster ID: 58e5d31b2aa04d2cac4840bd3e3505b1
I20260924 03:20:31.003821 3790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:31.023481 3790 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:31.023967 3790 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:31.033367 3790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 153d4d559bd844ae86d48b636fa7df6b: Generated new TSK 0
I20260924 03:20:31.034468 3790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:31.048396 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3708
W20260924 03:20:31.049858 3770 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53533
W20260924 03:20:31.050660 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3708: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:31.050812 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 3708
2026-09-24T03:20:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:31 dist-test-slave-vd0p krb5kdc[3806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:31 dist-test-slave-vd0p krb5kdc[3806](info): set up 2 sockets
Sep 24 03:20:31 dist-test-slave-vd0p krb5kdc[3806](info): commencing operation
krb5kdc: starting...
W20260924 03:20:32.218804 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:32 dist-test-slave-vd0p krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:32.460638 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:32.782876 3819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:32.783143 3819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:32.783362 3819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:32.790331 3819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:32.790462 3819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:32.790568 3819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:32.790685 3819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:32.790752 3819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:32.799968 3819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:32.801177 3819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:32.802474 3819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:32.814332 3827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:32.814306 3825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:32.814579 3824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:32.815233 3819 server_base.cc:1034] running on GCE node
I20260924 03:20:32.815984 3819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:32.817291 3819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:32.818512 3819 hybrid_clock.cc:648] HybridClock initialized: now 1790220032818465 us; error 72 us; skew 500 ppm
I20260924 03:20:32.818863 3819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:32 dist-test-slave-vd0p krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:32.821909 3819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:32.823370 3819 webserver.cc:460] Webserver started at http://127.0.0.1:41419/ using document root <none> and password file <none>
I20260924 03:20:32.824005 3819 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:32.824159 3819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:32.824443 3819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:32.827406 3819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "8f2ba8789be54db0885422e176114c9b"
format_stamp: "Formatted at 2026-09-24 03:20:32 on dist-test-slave-vd0p"
server_key: "3393d3e598050ab09f9be49380895b61"
server_key_iv: "62a9befc84f79f8d17f750765fbf97f8"
server_key_version: "encryptionkey@0"
I20260924 03:20:32.828117 3819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "8f2ba8789be54db0885422e176114c9b"
format_stamp: "Formatted at 2026-09-24 03:20:32 on dist-test-slave-vd0p"
server_key: "3393d3e598050ab09f9be49380895b61"
server_key_iv: "62a9befc84f79f8d17f750765fbf97f8"
server_key_version: "encryptionkey@0"
I20260924 03:20:32.832901 3819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 03:20:32.836090 3834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:32.837460 3819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:20:32.837677 3819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "8f2ba8789be54db0885422e176114c9b"
format_stamp: "Formatted at 2026-09-24 03:20:32 on dist-test-slave-vd0p"
server_key: "3393d3e598050ab09f9be49380895b61"
server_key_iv: "62a9befc84f79f8d17f750765fbf97f8"
server_key_version: "encryptionkey@0"
I20260924 03:20:32.837886 3819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:32.859404 3819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:32.863583 3819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:32.863898 3819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:32.883461 3819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40365
I20260924 03:20:32.883496 3885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40365 every 8 connection(s)
I20260924 03:20:32.884895 3819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:32.888559 3886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:32.893323 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 3819
I20260924 03:20:32.893499 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:32.894026 15246 external_mini_cluster.cc:1428] Setting key 19b9f9cfb22f209ab5b1ceb9aaa3714b
I20260924 03:20:32.899145 3886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b: Bootstrap starting.
I20260924 03:20:32.902701 3886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:32.903909 3886 log.cc:824] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:32.907153 3886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b: No bootstrap required, opened a new log
Sep 24 03:20:32 dist-test-slave-vd0p krb5kdc[3806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:20:32.915349 3886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2ba8789be54db0885422e176114c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260924 03:20:32.915731 3886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:32.915863 3886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2ba8789be54db0885422e176114c9b, State: Initialized, Role: FOLLOWER
I20260924 03:20:32.916483 3886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2ba8789be54db0885422e176114c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260924 03:20:32.916733 3886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:32.916890 3886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:32.917081 3886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:32.918089 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:32.895834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:49435 (local address 127.0.0.1:40365)
0924 03:20:32.896681 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:32.896692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:32.896741 (+ 49us) server_negotiation.cc:407] Connection header received
0924 03:20:32.898552 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:32.898573 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:32.898966 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:32.899359 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:32.901144 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:32.902549 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:32.904390 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:32.905427 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:32.905556 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:32.910464 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:32.910473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:32.910487 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:32.910529 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:32.913793 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:32.914590 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:32.914600 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:32.914606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:32.914772 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:32.915253 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:32.915257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:32.915259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:32.915753 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:32.916103 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:32.916592 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 03:20:32.916935 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":390,"threads_started":1}
I20260924 03:20:32.919508 3886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2ba8789be54db0885422e176114c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260924 03:20:32.919994 3886 leader_election.cc:302] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f2ba8789be54db0885422e176114c9b; no voters:
I20260924 03:20:32.920663 3886 leader_election.cc:288] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:32.920771 3891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:32.921854 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:32.923112 3886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:32.926859 3891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [term 1 LEADER]: Becoming Leader. State: Replica: 8f2ba8789be54db0885422e176114c9b, State: Running, Role: LEADER
I20260924 03:20:32.927901 3891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2ba8789be54db0885422e176114c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260924 03:20:32.931108 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f2ba8789be54db0885422e176114c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2ba8789be54db0885422e176114c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } } }
I20260924 03:20:32.931628 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:32.933740 3893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f2ba8789be54db0885422e176114c9b. Latest consensus state: current_term: 1 leader_uuid: "8f2ba8789be54db0885422e176114c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2ba8789be54db0885422e176114c9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } } }
I20260924 03:20:32.934773 3893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:32.939291 3909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:32.944023 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:32.926417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:54891 (local address 127.0.0.1:40365)
0924 03:20:32.926629 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:32.926637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:32.927456 (+ 819us) server_negotiation.cc:407] Connection header received
0924 03:20:32.927651 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:32.927658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:32.927771 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:32.927872 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:32.929805 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:32.931037 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:32.933071 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:32.934017 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:32.934169 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:32.937387 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:32.937395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:32.937403 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:32.937430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:32.940668 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:32.941591 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:32.941596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:32.941599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:32.941748 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:32.942309 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:32.942314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:32.942317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:32.942795 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:32.943094 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:32.943488 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 03:20:32.943663 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:20:32.944468 3909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:32.952221 3909 catalog_manager.cc:1347] Generated new cluster ID: e4c66b7f848349b89abc7598de9c65d3
I20260924 03:20:32.952342 3909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:32.970857 3909 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:32.972579 3909 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:32.979036 3909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f2ba8789be54db0885422e176114c9b: Generated new TSK 0
I20260924 03:20:32.979928 3909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:32.989280 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3819
I20260924 03:20:32.990392 3881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:33.078713 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 3819
W20260924 03:20:33.095225 3804 connection.cc:535] client connection to 127.0.0.1:40365 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40365): Connection reset by peer (error 104)
2026-09-24T03:20:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:33 dist-test-slave-vd0p krb5kdc[3919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:33 dist-test-slave-vd0p krb5kdc[3919](info): set up 2 sockets
Sep 24 03:20:33 dist-test-slave-vd0p krb5kdc[3919](info): commencing operation
krb5kdc: starting...
W20260924 03:20:34.170526 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:34 dist-test-slave-vd0p krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:34.432889 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:34.754382 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:34.754647 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:34.754796 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:34.761799 3932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:34.761972 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:34.762077 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:34.762188 3932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:34.762259 3932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:34.771788 3932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:34.772984 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:34.774263 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:34.785983 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:34.785983 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:34.786515 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:34.787109 3932 server_base.cc:1034] running on GCE node
I20260924 03:20:34.787891 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:34.789285 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:34.790473 3932 hybrid_clock.cc:648] HybridClock initialized: now 1790220034790415 us; error 49 us; skew 500 ppm
I20260924 03:20:34.790823 3932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:34 dist-test-slave-vd0p krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:34.793785 3932 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:34.795282 3932 webserver.cc:460] Webserver started at http://127.0.0.1:44771/ using document root <none> and password file <none>
I20260924 03:20:34.795928 3932 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:34.796078 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:34.796391 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:34.799347 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e"
format_stamp: "Formatted at 2026-09-24 03:20:34 on dist-test-slave-vd0p"
server_key: "c9869d44530e8d974e293fc16349b8b4"
server_key_iv: "aa2b845c1bf5983949d7bae32e81f918"
server_key_version: "encryptionkey@0"
I20260924 03:20:34.800093 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e"
format_stamp: "Formatted at 2026-09-24 03:20:34 on dist-test-slave-vd0p"
server_key: "c9869d44530e8d974e293fc16349b8b4"
server_key_iv: "aa2b845c1bf5983949d7bae32e81f918"
server_key_version: "encryptionkey@0"
I20260924 03:20:34.804965 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 03:20:34.808234 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:34.809500 3932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 03:20:34.809675 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e"
format_stamp: "Formatted at 2026-09-24 03:20:34 on dist-test-slave-vd0p"
server_key: "c9869d44530e8d974e293fc16349b8b4"
server_key_iv: "aa2b845c1bf5983949d7bae32e81f918"
server_key_version: "encryptionkey@0"
I20260924 03:20:34.809868 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:34.845319 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:34.849422 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:34.849713 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:34.867341 3932 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36553
I20260924 03:20:34.867365 3998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36553 every 8 connection(s)
I20260924 03:20:34.868651 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:34.872287 3999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:34.876809 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 3932
I20260924 03:20:34.877012 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:34.877533 15246 external_mini_cluster.cc:1428] Setting key e3acb76e7924a7bd640315eb4963929e
I20260924 03:20:34.883385 3999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e: Bootstrap starting.
I20260924 03:20:34.886795 3999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:34.888111 3999 log.cc:824] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:34.890940 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:34.879293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:52771 (local address 127.0.0.1:36553)
0924 03:20:34.880035 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:34.880046 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:34.880087 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:20:34.881918 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:34.881937 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:34.882389 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:34.882692 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:34.883267 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:34.884529 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:34.886402 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:34.887165 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:34.887300 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:34.888517 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:34.888524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:34.888539 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:34.889326 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:34.889430 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:34.889467 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:34.889585 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:34.889871 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 03:20:34.890074 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":337,"threads_started":1}
I20260924 03:20:34.891153 3999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e: No bootstrap required, opened a new log
I20260924 03:20:34.894443 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:34.900547 3999 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260924 03:20:34.900980 3999 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:34.901115 3999 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e8b20bb9c64a17a13fdf90e0c6045e, State: Initialized, Role: FOLLOWER
I20260924 03:20:34.901723 3999 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260924 03:20:34.902050 3999 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:34.902202 3999 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:34.902385 3999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:34.905128 3999 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260924 03:20:34.905699 3999 leader_election.cc:302] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4e8b20bb9c64a17a13fdf90e0c6045e; no voters:
I20260924 03:20:34.906209 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:34.899439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:32857 (local address 127.0.0.1:36553)
0924 03:20:34.899651 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:34.899658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:34.900135 (+ 477us) server_negotiation.cc:407] Connection header received
0924 03:20:34.900248 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:34.900253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:34.900330 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:34.900435 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:34.900891 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:34.902060 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:34.903475 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:34.904184 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:34.904301 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:34.905408 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:34.905414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:34.905418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:34.905438 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:34.905511 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:34.905531 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:34.905638 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:34.905805 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 03:20:34.905855 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:20:34.906488 3999 leader_election.cc:288] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:34.906657 4009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:34.907303 4009 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [term 1 LEADER]: Becoming Leader. State: Replica: c4e8b20bb9c64a17a13fdf90e0c6045e, State: Running, Role: LEADER
I20260924 03:20:34.907915 4009 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260924 03:20:34.909428 3999 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:34.910789 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e8b20bb9c64a17a13fdf90e0c6045e. Latest consensus state: current_term: 1 leader_uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260924 03:20:34.910818 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e8b20bb9c64a17a13fdf90e0c6045e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260924 03:20:34.911270 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:34.911299 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:34.912453 4014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:34.917387 4014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:34.927385 4014 catalog_manager.cc:1347] Generated new cluster ID: a342ed68f83e4ec1811a619c788b5c5e
I20260924 03:20:34.927510 4014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:34.938673 4014 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:34.940403 4014 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:34.951571 4014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4e8b20bb9c64a17a13fdf90e0c6045e: Generated new TSK 0
I20260924 03:20:34.952661 4014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:34.963882 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3932
W20260924 03:20:34.964918 3994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.163:52771
W20260924 03:20:34.965620 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 3932: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:34.965772 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 3932
W20260924 03:20:34.979055 3917 connection.cc:535] client connection to 127.0.0.1:36553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36553): Connection reset by peer (error 104)
2026-09-24T03:20:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:35 dist-test-slave-vd0p krb5kdc[4030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:35 dist-test-slave-vd0p krb5kdc[4030](info): set up 2 sockets
Sep 24 03:20:35 dist-test-slave-vd0p krb5kdc[4030](info): commencing operation
krb5kdc: starting...
W20260924 03:20:36.066363 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:36 dist-test-slave-vd0p krb5kdc[4030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:36.313753 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:36.643199 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:36.643580 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:36.643759 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:36.651186 4043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:36.651356 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:36.651471 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:36.651584 4043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:36.651683 4043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:36.661585 4043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:36.663014 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:36.664364 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:36.676297 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:36.676272 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:36.676222 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:36.677383 4043 server_base.cc:1034] running on GCE node
I20260924 03:20:36.678318 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:36.679749 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:36.680982 4043 hybrid_clock.cc:648] HybridClock initialized: now 1790220036680945 us; error 48 us; skew 500 ppm
I20260924 03:20:36.681341 4043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:36 dist-test-slave-vd0p krb5kdc[4030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:36.688968 4043 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:36.690629 4043 webserver.cc:460] Webserver started at http://127.0.0.1:41423/ using document root <none> and password file <none>
I20260924 03:20:36.691339 4043 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:36.691488 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:36.691802 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:36.694802 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "0eb6e2b30a34436d8440bd273fea0c67"
format_stamp: "Formatted at 2026-09-24 03:20:36 on dist-test-slave-vd0p"
server_key: "d44f4ccc51afe4bdee4948f75bf78984"
server_key_iv: "fdf4d79b96c1ef482b0c6eb1cc6f963b"
server_key_version: "encryptionkey@0"
I20260924 03:20:36.695557 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "0eb6e2b30a34436d8440bd273fea0c67"
format_stamp: "Formatted at 2026-09-24 03:20:36 on dist-test-slave-vd0p"
server_key: "d44f4ccc51afe4bdee4948f75bf78984"
server_key_iv: "fdf4d79b96c1ef482b0c6eb1cc6f963b"
server_key_version: "encryptionkey@0"
I20260924 03:20:36.700619 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:20:36.704121 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:36.705415 4043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:20:36.705607 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "0eb6e2b30a34436d8440bd273fea0c67"
format_stamp: "Formatted at 2026-09-24 03:20:36 on dist-test-slave-vd0p"
server_key: "d44f4ccc51afe4bdee4948f75bf78984"
server_key_iv: "fdf4d79b96c1ef482b0c6eb1cc6f963b"
server_key_version: "encryptionkey@0"
I20260924 03:20:36.705827 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:36.711624 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:36.715648 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:36.715971 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:36.734251 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34269
I20260924 03:20:36.734308 4109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34269 every 8 connection(s)
I20260924 03:20:36.735690 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:36.739696 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4043
I20260924 03:20:36.739737 4110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:36.739876 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:36.740429 15246 external_mini_cluster.cc:1428] Setting key fe6566e67b85ce97c46362dd71dda3ae
I20260924 03:20:36.747814 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:36.742252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:40033 (local address 127.0.0.1:34269)
0924 03:20:36.743101 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:36.743113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:36.743156 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:20:36.744592 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:36.744612 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:36.744941 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:36.745264 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:36.746331 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:36.746337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:36.746350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:36.746367 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:36.746420 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:36.746440 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:36.746609 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:36.746713 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 03:20:36.746941 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":405,"threads_started":1}
I20260924 03:20:36.751315 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:36.753247 4110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67: Bootstrap starting.
I20260924 03:20:36.757242 4110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:36.758710 4110 log.cc:824] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:36.759128 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:36.756483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:37815 (local address 127.0.0.1:34269)
0924 03:20:36.756673 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:36.756679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:36.756992 (+ 313us) server_negotiation.cc:407] Connection header received
0924 03:20:36.757192 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:36.757196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:36.757245 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:36.757345 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:36.758361 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:36.758365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:36.758368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:36.758382 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:36.758417 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:36.758430 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:36.758518 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:36.758667 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 03:20:36.758718 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:20:36.761621 4110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67: No bootstrap required, opened a new log
I20260924 03:20:36.768712 4110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb6e2b30a34436d8440bd273fea0c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } }
I20260924 03:20:36.769114 4110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:36.769218 4110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb6e2b30a34436d8440bd273fea0c67, State: Initialized, Role: FOLLOWER
I20260924 03:20:36.769783 4110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb6e2b30a34436d8440bd273fea0c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } }
I20260924 03:20:36.770048 4110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:36.770191 4110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:36.770364 4110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:36.772439 4110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb6e2b30a34436d8440bd273fea0c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } }
I20260924 03:20:36.773020 4110 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb6e2b30a34436d8440bd273fea0c67; no voters:
I20260924 03:20:36.773551 4110 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:36.773690 4120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:36.774335 4120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb6e2b30a34436d8440bd273fea0c67, State: Running, Role: LEADER
I20260924 03:20:36.774837 4120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb6e2b30a34436d8440bd273fea0c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } }
I20260924 03:20:36.775584 4110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:36.777444 4121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb6e2b30a34436d8440bd273fea0c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb6e2b30a34436d8440bd273fea0c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } } }
I20260924 03:20:36.777925 4121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:36.777418 4122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb6e2b30a34436d8440bd273fea0c67. Latest consensus state: current_term: 1 leader_uuid: "0eb6e2b30a34436d8440bd273fea0c67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb6e2b30a34436d8440bd273fea0c67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34269 } } }
I20260924 03:20:36.778210 4122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:36.780543 4126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:36.786226 4126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:36.795207 4126 catalog_manager.cc:1347] Generated new cluster ID: 16be25851cdd494ab5ab07b8f5aecedc
I20260924 03:20:36.795323 4126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:36.833036 4126 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:36.833487 4126 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:36.843219 4126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb6e2b30a34436d8440bd273fea0c67: Generated new TSK 0
I20260924 03:20:36.844090 4126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:36.857071 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4043
W20260924 03:20:36.858616 4105 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.163:40033
W20260924 03:20:36.859427 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4043: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:36.859581 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4043
2026-09-24T03:20:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:36 dist-test-slave-vd0p krb5kdc[4141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:36 dist-test-slave-vd0p krb5kdc[4141](info): set up 2 sockets
Sep 24 03:20:36 dist-test-slave-vd0p krb5kdc[4141](info): commencing operation
krb5kdc: starting...
W20260924 03:20:37.950295 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:38 dist-test-slave-vd0p krb5kdc[4141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:38.188862 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:38.505436 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:38.505712 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:38.505831 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:38.513099 4154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:38.513208 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:38.513254 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:38.513339 4154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:38.513410 4154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:38.523365 4154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:38.524677 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:38.526070 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:38.537699 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:38.537658 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:38.537742 4162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:38.538517 4154 server_base.cc:1034] running on GCE node
I20260924 03:20:38.539311 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:38.540732 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:38.541994 4154 hybrid_clock.cc:648] HybridClock initialized: now 1790220038541895 us; error 70 us; skew 500 ppm
I20260924 03:20:38.542358 4154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:38 dist-test-slave-vd0p krb5kdc[4141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:38.545324 4154 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:38.546655 4154 webserver.cc:460] Webserver started at http://127.0.0.1:43313/ using document root <none> and password file <none>
I20260924 03:20:38.547293 4154 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:38.547431 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:38.547708 4154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:38.550510 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "41f8f912814f429e89d6c18eb04b2aba"
format_stamp: "Formatted at 2026-09-24 03:20:38 on dist-test-slave-vd0p"
server_key: "fb642771712a0d233779a9719f929e71"
server_key_iv: "41036dda7decd362256b10e817504c19"
server_key_version: "encryptionkey@0"
I20260924 03:20:38.551326 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "41f8f912814f429e89d6c18eb04b2aba"
format_stamp: "Formatted at 2026-09-24 03:20:38 on dist-test-slave-vd0p"
server_key: "fb642771712a0d233779a9719f929e71"
server_key_iv: "41036dda7decd362256b10e817504c19"
server_key_version: "encryptionkey@0"
I20260924 03:20:38.556262 4154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 03:20:38.559492 4169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:38.560700 4154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 03:20:38.560889 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "41f8f912814f429e89d6c18eb04b2aba"
format_stamp: "Formatted at 2026-09-24 03:20:38 on dist-test-slave-vd0p"
server_key: "fb642771712a0d233779a9719f929e71"
server_key_iv: "41036dda7decd362256b10e817504c19"
server_key_version: "encryptionkey@0"
I20260924 03:20:38.561095 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:38.575212 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:38.578842 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:38.579058 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:38.600188 4154 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45911
I20260924 03:20:38.600241 4220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45911 every 8 connection(s)
I20260924 03:20:38.602089 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:38.607405 4221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:38.612468 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4154
I20260924 03:20:38.612675 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:38.613299 15246 external_mini_cluster.cc:1428] Setting key d14e0d5b5b0027091d53835bb5b8b45b
I20260924 03:20:38.627941 4221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba: Bootstrap starting.
Sep 24 03:20:38 dist-test-slave-vd0p krb5kdc[4141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:20:38.632571 4221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:38.634202 4221 log.cc:824] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:38.637970 4221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba: No bootstrap required, opened a new log
I20260924 03:20:38.641048 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:38.615155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:56527 (local address 127.0.0.1:45911)
0924 03:20:38.615884 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:38.615912 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:38.615985 (+ 73us) server_negotiation.cc:407] Connection header received
0924 03:20:38.618424 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:38.618455 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:38.618987 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:38.619418 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:38.621933 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:38.624142 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:38.626221 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:38.626932 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:38.627132 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:38.631995 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:38.632003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:38.632018 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:38.632057 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:38.635333 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:38.636101 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:38.636127 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:38.636142 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:38.636382 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:38.636816 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:38.636821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:38.636824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:38.637510 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:38.637833 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:38.639332 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0924 03:20:38.639669 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
I20260924 03:20:38.644838 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:38.647986 4221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f8f912814f429e89d6c18eb04b2aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260924 03:20:38.648504 4221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:38.648629 4221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f8f912814f429e89d6c18eb04b2aba, State: Initialized, Role: FOLLOWER
I20260924 03:20:38.649266 4221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f8f912814f429e89d6c18eb04b2aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260924 03:20:38.649545 4221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:38.649689 4221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:38.649871 4221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:38.652326 4221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f8f912814f429e89d6c18eb04b2aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260924 03:20:38.652866 4221 leader_election.cc:302] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f8f912814f429e89d6c18eb04b2aba; no voters:
I20260924 03:20:38.653558 4221 leader_election.cc:288] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:38.653658 4231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:38.656406 4221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:38.660588 4231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [term 1 LEADER]: Becoming Leader. State: Replica: 41f8f912814f429e89d6c18eb04b2aba, State: Running, Role: LEADER
I20260924 03:20:38.661343 4231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f8f912814f429e89d6c18eb04b2aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260924 03:20:38.664846 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:38.650180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:56493 (local address 127.0.0.1:45911)
0924 03:20:38.650335 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:38.650361 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:38.650785 (+ 424us) server_negotiation.cc:407] Connection header received
0924 03:20:38.650979 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:38.650988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:38.651066 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:38.651171 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:38.652651 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:38.653474 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:38.655197 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:38.655621 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:38.655744 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:38.658167 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:38.658179 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:38.658183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:38.658220 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:38.661311 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:38.662228 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:38.662235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:38.662239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:38.662377 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:38.662874 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:38.662883 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:38.662885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:38.663236 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:38.663560 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:38.664227 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 03:20:38.664466 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:20:38.664721 4232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f8f912814f429e89d6c18eb04b2aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f8f912814f429e89d6c18eb04b2aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260924 03:20:38.665264 4232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:38.670152 4233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f8f912814f429e89d6c18eb04b2aba. Latest consensus state: current_term: 1 leader_uuid: "41f8f912814f429e89d6c18eb04b2aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f8f912814f429e89d6c18eb04b2aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260924 03:20:38.670535 4233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:38.676908 4240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:38.687701 4240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:38.697568 4240 catalog_manager.cc:1347] Generated new cluster ID: dccf6fe008bd4c4a8134ee59cf438bc4
I20260924 03:20:38.697695 4240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:38.710553 4240 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:38.711838 4240 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:38.726055 4240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f8f912814f429e89d6c18eb04b2aba: Generated new TSK 0
I20260924 03:20:38.726873 4240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:38.736717 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4154
I20260924 03:20:38.737890 4216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:38.830067 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4154
W20260924 03:20:38.847186 4139 connection.cc:535] client connection to 127.0.0.1:45911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45911): Connection reset by peer (error 104)
2026-09-24T03:20:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1970 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:38 dist-test-slave-vd0p krb5kdc[4254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:38 dist-test-slave-vd0p krb5kdc[4254](info): set up 2 sockets
Sep 24 03:20:38 dist-test-slave-vd0p krb5kdc[4254](info): commencing operation
krb5kdc: starting...
W20260924 03:20:39.930395 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:40 dist-test-slave-vd0p krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:40.177211 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:40.501166 4267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:40.501477 4267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:40.501579 4267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:40.508889 4267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:40.509001 4267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:40.509142 4267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:40.509253 4267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:40.509337 4267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:40.520103 4267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:40.521482 4267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:40.522851 4267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:40.534806 4272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:40.534869 4275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:40.534893 4273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:40.535696 4267 server_base.cc:1034] running on GCE node
I20260924 03:20:40.536501 4267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:40.537890 4267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:40.539143 4267 hybrid_clock.cc:648] HybridClock initialized: now 1790220040539099 us; error 41 us; skew 500 ppm
I20260924 03:20:40.539489 4267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:40 dist-test-slave-vd0p krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:40.542743 4267 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:40.544237 4267 webserver.cc:460] Webserver started at http://127.0.0.1:46105/ using document root <none> and password file <none>
I20260924 03:20:40.544931 4267 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:40.545037 4267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:40.545275 4267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:40.548180 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "6bf92208cc154fb1a7b0b13dd14aa997"
format_stamp: "Formatted at 2026-09-24 03:20:40 on dist-test-slave-vd0p"
server_key: "dbaf3bbea245c6e4cfa0915cf90d4afb"
server_key_iv: "43e8213bb002e11b51ce0a51e972c855"
server_key_version: "encryptionkey@0"
I20260924 03:20:40.548872 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "6bf92208cc154fb1a7b0b13dd14aa997"
format_stamp: "Formatted at 2026-09-24 03:20:40 on dist-test-slave-vd0p"
server_key: "dbaf3bbea245c6e4cfa0915cf90d4afb"
server_key_iv: "43e8213bb002e11b51ce0a51e972c855"
server_key_version: "encryptionkey@0"
I20260924 03:20:40.553843 4267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 03:20:40.557199 4282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:40.558589 4267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:40.558773 4267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "6bf92208cc154fb1a7b0b13dd14aa997"
format_stamp: "Formatted at 2026-09-24 03:20:40 on dist-test-slave-vd0p"
server_key: "dbaf3bbea245c6e4cfa0915cf90d4afb"
server_key_iv: "43e8213bb002e11b51ce0a51e972c855"
server_key_version: "encryptionkey@0"
I20260924 03:20:40.558957 4267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:40.578692 4267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:40.583211 4267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:40.583505 4267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:40.601933 4267 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36591
I20260924 03:20:40.601992 4333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36591 every 8 connection(s)
I20260924 03:20:40.603412 4267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:40.607239 4334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:40.611235 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4267
I20260924 03:20:40.611426 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:40.611851 15246 external_mini_cluster.cc:1428] Setting key f1851194886feccee58abb76d32760d1
I20260924 03:20:40.620951 4334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997: Bootstrap starting.
I20260924 03:20:40.624926 4334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:40.626343 4334 log.cc:824] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:40.626963 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:40.613777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:45871 (local address 127.0.0.1:36591)
0924 03:20:40.614616 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:40.614633 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:40.614692 (+ 59us) server_negotiation.cc:407] Connection header received
0924 03:20:40.616451 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:40.616481 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:40.617075 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:40.617523 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:40.618121 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:40.620107 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:40.621534 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:40.622546 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:40.622782 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:40.623947 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:40.623958 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:40.623976 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:40.624700 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:40.624809 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:40.624845 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:40.625002 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:40.625356 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 03:20:40.625551 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":382,"threads_started":1}
I20260924 03:20:40.629740 4334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997: No bootstrap required, opened a new log
I20260924 03:20:40.631234 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:40.638528 4334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf92208cc154fb1a7b0b13dd14aa997" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36591 } }
I20260924 03:20:40.638906 4334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:40.639015 4334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf92208cc154fb1a7b0b13dd14aa997, State: Initialized, Role: FOLLOWER
I20260924 03:20:40.639726 4334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf92208cc154fb1a7b0b13dd14aa997" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36591 } }
I20260924 03:20:40.640022 4334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:40.640180 4334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:40.640365 4334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:40.642416 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:40.635667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:60443 (local address 127.0.0.1:36591)
0924 03:20:40.635840 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:40.635846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:40.636204 (+ 358us) server_negotiation.cc:407] Connection header received
0924 03:20:40.636336 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:40.636342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:40.636434 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:40.636544 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:40.637023 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:40.638165 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:40.639544 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:40.640231 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:40.640367 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:40.641512 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:40.641518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:40.641524 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:40.641545 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:40.641599 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:40.641618 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:40.641746 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:40.641940 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 03:20:40.642023 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:20:40.643986 4334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf92208cc154fb1a7b0b13dd14aa997" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36591 } }
I20260924 03:20:40.644580 4334 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf92208cc154fb1a7b0b13dd14aa997; no voters:
I20260924 03:20:40.645224 4334 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:40.645442 4344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:40.646394 4344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [term 1 LEADER]: Becoming Leader. State: Replica: 6bf92208cc154fb1a7b0b13dd14aa997, State: Running, Role: LEADER
I20260924 03:20:40.647082 4344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf92208cc154fb1a7b0b13dd14aa997" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36591 } }
I20260924 03:20:40.647691 4334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:40.650523 4345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf92208cc154fb1a7b0b13dd14aa997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf92208cc154fb1a7b0b13dd14aa997" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36591 } } }
I20260924 03:20:40.651136 4345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:40.651201 4346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf92208cc154fb1a7b0b13dd14aa997. Latest consensus state: current_term: 1 leader_uuid: "6bf92208cc154fb1a7b0b13dd14aa997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf92208cc154fb1a7b0b13dd14aa997" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36591 } } }
I20260924 03:20:40.651762 4346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:40.652650 4352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:40.665935 4352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:40.675264 4352 catalog_manager.cc:1347] Generated new cluster ID: 30433b7cbd8c43aebf0a8cd3839be966
I20260924 03:20:40.675397 4352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:40.697290 4352 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:40.699157 4352 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:40.716041 4352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf92208cc154fb1a7b0b13dd14aa997: Generated new TSK 0
I20260924 03:20:40.716905 4352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:40.731791 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4267
W20260924 03:20:40.732851 4329 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.163:45871
W20260924 03:20:40.733575 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4267: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:40.733732 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4267
W20260924 03:20:40.750700 4252 connection.cc:535] client connection to 127.0.0.1:36591 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36591): Connection reset by peer (error 104)
2026-09-24T03:20:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:40 dist-test-slave-vd0p krb5kdc[4365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:40 dist-test-slave-vd0p krb5kdc[4365](info): set up 2 sockets
Sep 24 03:20:40 dist-test-slave-vd0p krb5kdc[4365](info): commencing operation
krb5kdc: starting...
W20260924 03:20:41.830487 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:41 dist-test-slave-vd0p krb5kdc[4365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:20:42.072834 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:42.396381 4378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:42.396651 4378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:42.396747 4378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:42.403901 4378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:42.404024 4378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:42.404101 4378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:42.404194 4378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:42.404249 4378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:42.413615 4378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:42.414881 4378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:42.416122 4378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:42.428889 4383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:42.428860 4384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:42.429015 4386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:42.429523 4378 server_base.cc:1034] running on GCE node
I20260924 03:20:42.430383 4378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:42.431843 4378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:42.433094 4378 hybrid_clock.cc:648] HybridClock initialized: now 1790220042433027 us; error 60 us; skew 500 ppm
I20260924 03:20:42.433465 4378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:42 dist-test-slave-vd0p krb5kdc[4365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:42.440559 4378 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:20:42.442199 4378 webserver.cc:460] Webserver started at http://127.0.0.1:41439/ using document root <none> and password file <none>
I20260924 03:20:42.442941 4378 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:42.443090 4378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:42.443405 4378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:42.446614 4378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "cb8e5cdb4c164106b5cea72728a2657d"
format_stamp: "Formatted at 2026-09-24 03:20:42 on dist-test-slave-vd0p"
server_key: "5d16f5fe522f8e315cb430058c948672"
server_key_iv: "67b1449c1ec3e62710aa87049194d4e3"
server_key_version: "encryptionkey@0"
I20260924 03:20:42.447423 4378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "cb8e5cdb4c164106b5cea72728a2657d"
format_stamp: "Formatted at 2026-09-24 03:20:42 on dist-test-slave-vd0p"
server_key: "5d16f5fe522f8e315cb430058c948672"
server_key_iv: "67b1449c1ec3e62710aa87049194d4e3"
server_key_version: "encryptionkey@0"
I20260924 03:20:42.453217 4378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:20:42.456859 4393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:42.458519 4378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:42.458740 4378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "cb8e5cdb4c164106b5cea72728a2657d"
format_stamp: "Formatted at 2026-09-24 03:20:42 on dist-test-slave-vd0p"
server_key: "5d16f5fe522f8e315cb430058c948672"
server_key_iv: "67b1449c1ec3e62710aa87049194d4e3"
server_key_version: "encryptionkey@0"
I20260924 03:20:42.458948 4378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:42.464721 4378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:42.469120 4378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:42.469388 4378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:42.488168 4378 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35307
I20260924 03:20:42.488209 4444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35307 every 8 connection(s)
I20260924 03:20:42.489580 4378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:42.493166 4445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:42.496042 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4378
I20260924 03:20:42.496210 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:42.496742 15246 external_mini_cluster.cc:1428] Setting key 773cdfd47805a41b769e1a2fa6beac58
I20260924 03:20:42.504277 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:42.498568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:58681 (local address 127.0.0.1:35307)
0924 03:20:42.499317 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:42.499328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:42.499366 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:20:42.500908 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:42.500929 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:42.501255 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:42.501658 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:42.502741 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:42.502746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:42.502758 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:42.502771 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:42.502822 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:42.502843 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:42.503097 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:42.503246 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 03:20:42.503422 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":309,"threads_started":1}
I20260924 03:20:42.505035 4445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d: Bootstrap starting.
I20260924 03:20:42.508054 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:42.508752 4445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:42.509774 4445 log.cc:824] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:42.512890 4445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d: No bootstrap required, opened a new log
I20260924 03:20:42.515014 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:42.512536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.163:52047 (local address 127.0.0.1:35307)
0924 03:20:42.512709 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:42.512714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:42.513139 (+ 425us) server_negotiation.cc:407] Connection header received
0924 03:20:42.513268 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:42.513273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:42.513342 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:42.513466 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:42.514372 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:42.514377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:42.514381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:42.514398 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:42.514445 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:42.514468 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:42.514587 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:42.514638 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:20:42.514695 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:20:42.521654 4445 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8e5cdb4c164106b5cea72728a2657d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35307 } }
I20260924 03:20:42.522080 4445 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:42.522161 4445 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8e5cdb4c164106b5cea72728a2657d, State: Initialized, Role: FOLLOWER
I20260924 03:20:42.522647 4445 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8e5cdb4c164106b5cea72728a2657d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35307 } }
I20260924 03:20:42.522897 4445 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:42.523000 4445 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:42.523120 4445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:42.525671 4445 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8e5cdb4c164106b5cea72728a2657d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35307 } }
I20260924 03:20:42.526182 4445 leader_election.cc:302] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8e5cdb4c164106b5cea72728a2657d; no voters:
I20260924 03:20:42.526724 4445 leader_election.cc:288] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:42.526880 4455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:42.527556 4455 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [term 1 LEADER]: Becoming Leader. State: Replica: cb8e5cdb4c164106b5cea72728a2657d, State: Running, Role: LEADER
I20260924 03:20:42.528118 4455 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8e5cdb4c164106b5cea72728a2657d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35307 } }
I20260924 03:20:42.528862 4445 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:42.531090 4457 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb8e5cdb4c164106b5cea72728a2657d. Latest consensus state: current_term: 1 leader_uuid: "cb8e5cdb4c164106b5cea72728a2657d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8e5cdb4c164106b5cea72728a2657d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35307 } } }
I20260924 03:20:42.531690 4457 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:42.531558 4456 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb8e5cdb4c164106b5cea72728a2657d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8e5cdb4c164106b5cea72728a2657d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35307 } } }
I20260924 03:20:42.531848 4456 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:42.532644 4464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:42.538125 4464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:42.547291 4464 catalog_manager.cc:1347] Generated new cluster ID: 5c41a8e7c24f4a2399225d3feab66bd6
I20260924 03:20:42.547426 4464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:42.558171 4464 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:42.558658 4464 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:42.568162 4464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb8e5cdb4c164106b5cea72728a2657d: Generated new TSK 0
I20260924 03:20:42.568899 4464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:42.572834 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4378
W20260924 03:20:42.574388 4440 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.163:58681
W20260924 03:20:42.575060 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4378: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:42.575215 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4378
2026-09-24T03:20:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1841 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:42 dist-test-slave-vd0p krb5kdc[4476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:42 dist-test-slave-vd0p krb5kdc[4476](info): set up 2 sockets
Sep 24 03:20:42 dist-test-slave-vd0p krb5kdc[4476](info): commencing operation
krb5kdc: starting...
W20260924 03:20:43.674244 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:43 dist-test-slave-vd0p krb5kdc[4476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:43.921940 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:40393
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:40393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:44.248173 4489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:44.248440 4489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:44.248593 4489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:44.255736 4489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:44.255831 4489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:44.255942 4489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:44.256055 4489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:44.256140 4489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:44.265646 4489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:40393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:44.266930 4489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:44.268270 4489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:44.279551 4494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:44.279498 4497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:44.279585 4495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:44.280542 4489 server_base.cc:1034] running on GCE node
I20260924 03:20:44.281292 4489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:44.282662 4489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:44.283876 4489 hybrid_clock.cc:648] HybridClock initialized: now 1790220044283848 us; error 54 us; skew 500 ppm
I20260924 03:20:44.284226 4489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:44 dist-test-slave-vd0p krb5kdc[4476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220044, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:44.287235 4489 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:44.288687 4489 webserver.cc:460] Webserver started at http://127.14.227.190:39259/ using document root <none> and password file <none>
I20260924 03:20:44.289320 4489 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:44.289454 4489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:44.289721 4489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:44.292590 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "664b30c1e6ad42fc952c144d8c2975c9"
format_stamp: "Formatted at 2026-09-24 03:20:44 on dist-test-slave-vd0p"
server_key: "88bcffebce89352f7aa548d8b66b640f"
server_key_iv: "2cba92dd66fdd92fd7e4f32da948c136"
server_key_version: "encryptionkey@0"
I20260924 03:20:44.293329 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "664b30c1e6ad42fc952c144d8c2975c9"
format_stamp: "Formatted at 2026-09-24 03:20:44 on dist-test-slave-vd0p"
server_key: "88bcffebce89352f7aa548d8b66b640f"
server_key_iv: "2cba92dd66fdd92fd7e4f32da948c136"
server_key_version: "encryptionkey@0"
I20260924 03:20:44.298316 4489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 03:20:44.301608 4504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:44.302896 4489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:44.303079 4489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "664b30c1e6ad42fc952c144d8c2975c9"
format_stamp: "Formatted at 2026-09-24 03:20:44 on dist-test-slave-vd0p"
server_key: "88bcffebce89352f7aa548d8b66b640f"
server_key_iv: "2cba92dd66fdd92fd7e4f32da948c136"
server_key_version: "encryptionkey@0"
I20260924 03:20:44.303270 4489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:44.334327 4489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:44.338351 4489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:44.338635 4489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:44.357416 4489 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:40393
I20260924 03:20:44.357439 4555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:40393 every 8 connection(s)
I20260924 03:20:44.358814 4489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:44.362335 4556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:44.365563 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4489
I20260924 03:20:44.365751 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:44.366245 15246 external_mini_cluster.cc:1428] Setting key a296d5c1e4a31f05508f62f29c414e25
I20260924 03:20:44.373719 4556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9: Bootstrap starting.
I20260924 03:20:44.376662 4556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:44.377642 4556 log.cc:824] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9: Log is configured to *not* fsync() on all Append() calls
Sep 24 03:20:44 dist-test-slave-vd0p krb5kdc[4476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:44.380290 4556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9: No bootstrap required, opened a new log
I20260924 03:20:44.387406 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:44.368006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57197 (local address 127.14.227.190:40393)
0924 03:20:44.368783 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:44.368795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:44.368834 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:20:44.370508 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:44.370529 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:44.370908 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:44.371302 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:44.372679 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:44.374060 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:44.375284 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:44.375938 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:44.376124 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:44.380237 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:44.380243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:44.380256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:44.380293 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:44.383379 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:44.384112 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:44.384118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:44.384123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:44.384298 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:44.384740 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:44.384744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:44.384746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:44.385234 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:44.385589 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:44.386067 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 03:20:44.386423 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":356,"threads_started":1}
I20260924 03:20:44.388331 4556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664b30c1e6ad42fc952c144d8c2975c9" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40393 } }
I20260924 03:20:44.388777 4556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:44.388898 4556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664b30c1e6ad42fc952c144d8c2975c9, State: Initialized, Role: FOLLOWER
I20260924 03:20:44.389518 4556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664b30c1e6ad42fc952c144d8c2975c9" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40393 } }
I20260924 03:20:44.389806 4556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:44.389988 4556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:44.390167 4556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:44.391310 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:44.392612 4556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664b30c1e6ad42fc952c144d8c2975c9" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40393 } }
I20260924 03:20:44.393085 4556 leader_election.cc:302] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 664b30c1e6ad42fc952c144d8c2975c9; no voters:
I20260924 03:20:44.393716 4556 leader_election.cc:288] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:44.393822 4561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:44.395714 4561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [term 1 LEADER]: Becoming Leader. State: Replica: 664b30c1e6ad42fc952c144d8c2975c9, State: Running, Role: LEADER
I20260924 03:20:44.396353 4561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664b30c1e6ad42fc952c144d8c2975c9" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40393 } }
I20260924 03:20:44.397639 4556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:44.401598 4565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 664b30c1e6ad42fc952c144d8c2975c9. Latest consensus state: current_term: 1 leader_uuid: "664b30c1e6ad42fc952c144d8c2975c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664b30c1e6ad42fc952c144d8c2975c9" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40393 } } }
I20260924 03:20:44.401904 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "664b30c1e6ad42fc952c144d8c2975c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664b30c1e6ad42fc952c144d8c2975c9" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40393 } } }
I20260924 03:20:44.402235 4565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:44.402311 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:44.415992 4575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:44.423322 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:44.395663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34185 (local address 127.14.227.190:40393)
0924 03:20:44.395872 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:44.395880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:44.402824 (+ 6944us) server_negotiation.cc:407] Connection header received
0924 03:20:44.403283 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:44.403289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:44.403362 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:44.403451 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:44.405911 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:44.407450 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:44.410026 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:44.410931 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:44.411061 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:44.415200 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:44.415208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:44.415213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:44.415249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:44.419639 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:44.420584 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:44.420589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:44.420592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:44.420736 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:44.421299 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:44.421304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:44.421307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:44.421761 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:44.422022 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:44.422694 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 03:20:44.422898 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:20:44.423156 4575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:44.432545 4575 catalog_manager.cc:1347] Generated new cluster ID: 6947bcf9c2874d798b3ad12c5143a9c2
I20260924 03:20:44.432801 4575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:44.441427 4575 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:44.442741 4575 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:44.452167 4575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 664b30c1e6ad42fc952c144d8c2975c9: Generated new TSK 0
I20260924 03:20:44.452965 4575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:44.464394 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4489
I20260924 03:20:44.465504 4551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:44.554247 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4489
W20260924 03:20:44.570886 4474 connection.cc:535] client connection to 127.14.227.190:40393 recv error: Network error: failed to read from TLS socket (remote: 127.14.227.190:40393): Connection reset by peer (error 104)
2026-09-24T03:20:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:44 dist-test-slave-vd0p krb5kdc[4588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:44 dist-test-slave-vd0p krb5kdc[4588](info): set up 2 sockets
Sep 24 03:20:44 dist-test-slave-vd0p krb5kdc[4588](info): commencing operation
krb5kdc: starting...
W20260924 03:20:45.638441 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:45 dist-test-slave-vd0p krb5kdc[4588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:45.874818 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33061
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:37533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:46.195767 4601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:46.196031 4601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:46.196187 4601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:46.203289 4601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:46.203392 4601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:46.203500 4601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:46.203612 4601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:46.203692 4601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:46.213088 4601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:37533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:33061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:46.214377 4601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:46.215641 4601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:46.227375 4606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:46.227456 4609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:46.227432 4607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:46.228291 4601 server_base.cc:1034] running on GCE node
I20260924 03:20:46.229079 4601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:46.230489 4601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:46.231722 4601 hybrid_clock.cc:648] HybridClock initialized: now 1790220046231694 us; error 45 us; skew 500 ppm
I20260924 03:20:46.232066 4601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:46 dist-test-slave-vd0p krb5kdc[4588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220046, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:46.235109 4601 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:46.236590 4601 webserver.cc:460] Webserver started at http://127.14.227.190:45083/ using document root <none> and password file <none>
I20260924 03:20:46.237260 4601 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:46.237404 4601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:46.237699 4601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:46.240604 4601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "58442b8111064816906366b3ed929a91"
format_stamp: "Formatted at 2026-09-24 03:20:46 on dist-test-slave-vd0p"
server_key: "9ab599551c8f2350e4846114cc30dc94"
server_key_iv: "2106d6f49677e2f4db86aac14b02e27e"
server_key_version: "encryptionkey@0"
I20260924 03:20:46.241346 4601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "58442b8111064816906366b3ed929a91"
format_stamp: "Formatted at 2026-09-24 03:20:46 on dist-test-slave-vd0p"
server_key: "9ab599551c8f2350e4846114cc30dc94"
server_key_iv: "2106d6f49677e2f4db86aac14b02e27e"
server_key_version: "encryptionkey@0"
I20260924 03:20:46.246307 4601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 03:20:46.249469 4616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:46.250738 4601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:46.250916 4601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "58442b8111064816906366b3ed929a91"
format_stamp: "Formatted at 2026-09-24 03:20:46 on dist-test-slave-vd0p"
server_key: "9ab599551c8f2350e4846114cc30dc94"
server_key_iv: "2106d6f49677e2f4db86aac14b02e27e"
server_key_version: "encryptionkey@0"
I20260924 03:20:46.251111 4601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:46.269001 4601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:46.272763 4601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:46.273061 4601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:46.291074 4601 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:33061
I20260924 03:20:46.291085 4667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:33061 every 8 connection(s)
I20260924 03:20:46.292415 4601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:46.295971 4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:46.297801 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4601
I20260924 03:20:46.298036 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:46.298611 15246 external_mini_cluster.cc:1428] Setting key b09fb37f36a5097aceae4b3ee61af6be
I20260924 03:20:46.309522 4668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91: Bootstrap starting.
I20260924 03:20:46.311275 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:46.300279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55321 (local address 127.14.227.190:33061)
0924 03:20:46.301071 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:46.301082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:46.301125 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:20:46.302756 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:46.302780 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:46.303154 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:46.303455 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:46.303980 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:46.305221 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:46.307084 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:46.307742 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:46.307861 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:46.309041 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:46.309049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:46.309066 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:46.309612 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:46.309711 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:46.309737 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:46.309859 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:46.310189 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 03:20:46.310349 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":386,"threads_started":1}
I20260924 03:20:46.312662 4668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:46.313699 4668 log.cc:824] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:46.315085 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:46.316449 4668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91: No bootstrap required, opened a new log
I20260924 03:20:46.325318 4668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58442b8111064816906366b3ed929a91" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33061 } }
I20260924 03:20:46.325852 4668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:46.326077 4668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58442b8111064816906366b3ed929a91, State: Initialized, Role: FOLLOWER
I20260924 03:20:46.326401 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:46.319485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50667 (local address 127.14.227.190:33061)
0924 03:20:46.319673 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:46.319679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:46.320075 (+ 396us) server_negotiation.cc:407] Connection header received
0924 03:20:46.320231 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:46.320236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:46.320310 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:46.320421 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:46.320872 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:46.322154 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:46.323650 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:46.324344 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:46.324465 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:46.325501 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:46.325506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:46.325510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:46.325530 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:46.325573 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:46.325601 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:46.325720 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:46.325922 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 03:20:46.326031 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:20:46.326797 4668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58442b8111064816906366b3ed929a91" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33061 } }
I20260924 03:20:46.327109 4668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:46.327271 4668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:46.327454 4668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:46.329913 4668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58442b8111064816906366b3ed929a91" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33061 } }
I20260924 03:20:46.330554 4668 leader_election.cc:302] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58442b8111064816906366b3ed929a91; no voters:
I20260924 03:20:46.331302 4668 leader_election.cc:288] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:46.331421 4678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:46.332118 4678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [term 1 LEADER]: Becoming Leader. State: Replica: 58442b8111064816906366b3ed929a91, State: Running, Role: LEADER
I20260924 03:20:46.332625 4678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58442b8111064816906366b3ed929a91" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33061 } }
I20260924 03:20:46.333362 4668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:46.335528 4680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58442b8111064816906366b3ed929a91. Latest consensus state: current_term: 1 leader_uuid: "58442b8111064816906366b3ed929a91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58442b8111064816906366b3ed929a91" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33061 } } }
I20260924 03:20:46.335968 4680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:46.336570 4679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58442b8111064816906366b3ed929a91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58442b8111064816906366b3ed929a91" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 33061 } } }
I20260924 03:20:46.336939 4679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:46.337759 4683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:46.342777 4683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:46.351276 4683 catalog_manager.cc:1347] Generated new cluster ID: a3a0b80712a84b9b8d268556a4eeb7c2
I20260924 03:20:46.351531 4683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:46.370785 4683 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:46.372522 4683 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:46.387113 4683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58442b8111064816906366b3ed929a91: Generated new TSK 0
I20260924 03:20:46.388059 4683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:46.400717 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4601
W20260924 03:20:46.401831 4663 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55321
W20260924 03:20:46.402567 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4601: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:46.402717 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4601
W20260924 03:20:46.420094 4586 connection.cc:535] client connection to 127.14.227.190:33061 recv error: Network error: failed to read from TLS socket (remote: 127.14.227.190:33061): Connection reset by peer (error 104)
2026-09-24T03:20:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:46 dist-test-slave-vd0p krb5kdc[4699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:46 dist-test-slave-vd0p krb5kdc[4699](info): set up 2 sockets
Sep 24 03:20:46 dist-test-slave-vd0p krb5kdc[4699](info): commencing operation
krb5kdc: starting...
W20260924 03:20:47.502468 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:47 dist-test-slave-vd0p krb5kdc[4699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:47.751382 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:41995
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:41995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:48.071202 4712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:48.071580 4712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:48.071753 4712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:48.079066 4712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:48.079255 4712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:48.079360 4712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:48.079483 4712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:48.079551 4712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:48.090206 4712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:41995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:41995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:48.091560 4712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:48.092862 4712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:48.104650 4717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:48.104732 4718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:48.104694 4720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:48.105842 4712 server_base.cc:1034] running on GCE node
I20260924 03:20:48.106633 4712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:48.108058 4712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:48.109308 4712 hybrid_clock.cc:648] HybridClock initialized: now 1790220048109257 us; error 58 us; skew 500 ppm
I20260924 03:20:48.109705 4712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:48 dist-test-slave-vd0p krb5kdc[4699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220048, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:48.116402 4712 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:48.117810 4712 webserver.cc:460] Webserver started at http://127.14.227.190:38667/ using document root <none> and password file <none>
I20260924 03:20:48.118531 4712 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:48.118676 4712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:48.118973 4712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:48.121833 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21"
format_stamp: "Formatted at 2026-09-24 03:20:48 on dist-test-slave-vd0p"
server_key: "56c9477d30dd4912f9046a0754d3c8ed"
server_key_iv: "a60f9249221e879698d65b869a9f9699"
server_key_version: "encryptionkey@0"
I20260924 03:20:48.122661 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21"
format_stamp: "Formatted at 2026-09-24 03:20:48 on dist-test-slave-vd0p"
server_key: "56c9477d30dd4912f9046a0754d3c8ed"
server_key_iv: "a60f9249221e879698d65b869a9f9699"
server_key_version: "encryptionkey@0"
I20260924 03:20:48.127769 4712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 03:20:48.131114 4727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:48.132337 4712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 03:20:48.132517 4712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21"
format_stamp: "Formatted at 2026-09-24 03:20:48 on dist-test-slave-vd0p"
server_key: "56c9477d30dd4912f9046a0754d3c8ed"
server_key_iv: "a60f9249221e879698d65b869a9f9699"
server_key_version: "encryptionkey@0"
I20260924 03:20:48.132707 4712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:48.138353 4712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:48.142562 4712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:48.142856 4712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:48.161098 4712 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:41995
I20260924 03:20:48.161119 4778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:41995 every 8 connection(s)
I20260924 03:20:48.162575 4712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:48.164441 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4712
I20260924 03:20:48.164618 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:48.165236 15246 external_mini_cluster.cc:1428] Setting key 7ce36d571af76338d32e402d7ef9e2c7
I20260924 03:20:48.167430 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:48.173060 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:48.167186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53783 (local address 127.14.227.190:41995)
0924 03:20:48.167998 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:48.168011 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:48.168050 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:20:48.169634 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:48.169662 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:48.170069 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:48.170458 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:48.171494 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:48.171500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:48.171511 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:48.171525 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:48.171599 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:48.171623 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:48.171863 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:48.172002 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:20:48.172181 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":403,"threads_started":1}
I20260924 03:20:48.176620 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:48.179587 4779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21: Bootstrap starting.
I20260924 03:20:48.183338 4779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:48.183338 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:48.180854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44295 (local address 127.14.227.190:41995)
0924 03:20:48.181048 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:48.181056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:48.181430 (+ 374us) server_negotiation.cc:407] Connection header received
0924 03:20:48.181610 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:48.181615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:48.181666 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:48.181771 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:48.182712 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:48.182717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:48.182722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:48.182744 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:48.182781 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:48.182795 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:48.182913 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:48.183011 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:20:48.183066 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 03:20:48.184585 4779 log.cc:824] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:48.187575 4779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21: No bootstrap required, opened a new log
I20260924 03:20:48.195258 4779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 41995 } }
I20260924 03:20:48.195597 4779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:48.195685 4779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb6cf2f5dce44b9a01b0183e80d2b21, State: Initialized, Role: FOLLOWER
I20260924 03:20:48.196174 4779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 41995 } }
I20260924 03:20:48.196388 4779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:48.196491 4779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:48.196632 4779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:48.199115 4779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 41995 } }
I20260924 03:20:48.199589 4779 leader_election.cc:302] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bb6cf2f5dce44b9a01b0183e80d2b21; no voters:
I20260924 03:20:48.200181 4779 leader_election.cc:288] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:48.200352 4789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:48.201131 4789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb6cf2f5dce44b9a01b0183e80d2b21, State: Running, Role: LEADER
I20260924 03:20:48.201648 4789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 41995 } }
I20260924 03:20:48.202458 4779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:48.204516 4791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bb6cf2f5dce44b9a01b0183e80d2b21. Latest consensus state: current_term: 1 leader_uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 41995 } } }
I20260924 03:20:48.204872 4791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:48.205152 4790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb6cf2f5dce44b9a01b0183e80d2b21" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 41995 } } }
I20260924 03:20:48.205438 4790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:48.212201 4796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:48.218842 4796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:48.227828 4796 catalog_manager.cc:1347] Generated new cluster ID: fca8d867b5e14701b645d19f0c1a2eec
I20260924 03:20:48.227964 4796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:48.254642 4796 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:48.255101 4796 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:48.265715 4796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bb6cf2f5dce44b9a01b0183e80d2b21: Generated new TSK 0
I20260924 03:20:48.266752 4796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:48.283217 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4712
W20260924 03:20:48.284252 4774 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53783
W20260924 03:20:48.284932 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4712: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:48.285086 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4712
2026-09-24T03:20:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:48 dist-test-slave-vd0p krb5kdc[4810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:48 dist-test-slave-vd0p krb5kdc[4810](info): set up 2 sockets
Sep 24 03:20:48 dist-test-slave-vd0p krb5kdc[4810](info): commencing operation
krb5kdc: starting...
W20260924 03:20:49.390733 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:49 dist-test-slave-vd0p krb5kdc[4810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:49.647583 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:40501
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:40501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:49.970160 4823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:49.970511 4823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:49.970670 4823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:49.977871 4823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:49.978041 4823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:49.978150 4823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:49.978255 4823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:49.978329 4823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:49.987965 4823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:40501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:40501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:49.989218 4823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:49.990577 4823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:50.001971 4828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:50.002148 4829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:50.002125 4831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:50.003223 4823 server_base.cc:1034] running on GCE node
I20260924 03:20:50.003957 4823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:50.005332 4823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:50.006553 4823 hybrid_clock.cc:648] HybridClock initialized: now 1790220050006510 us; error 47 us; skew 500 ppm
I20260924 03:20:50.006888 4823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:50 dist-test-slave-vd0p krb5kdc[4810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220050, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:50.010031 4823 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:50.011505 4823 webserver.cc:460] Webserver started at http://127.14.227.190:36799/ using document root <none> and password file <none>
I20260924 03:20:50.012166 4823 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:50.012341 4823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:50.012631 4823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:50.015555 4823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "ca156326fa0f4ea6912079dcf350a7f1"
format_stamp: "Formatted at 2026-09-24 03:20:50 on dist-test-slave-vd0p"
server_key: "af72c00b03d2f4abfa5a2fadfd79d4c1"
server_key_iv: "cddbc00c2a87cdfa660ca9cd9b189b38"
server_key_version: "encryptionkey@0"
I20260924 03:20:50.016305 4823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "ca156326fa0f4ea6912079dcf350a7f1"
format_stamp: "Formatted at 2026-09-24 03:20:50 on dist-test-slave-vd0p"
server_key: "af72c00b03d2f4abfa5a2fadfd79d4c1"
server_key_iv: "cddbc00c2a87cdfa660ca9cd9b189b38"
server_key_version: "encryptionkey@0"
I20260924 03:20:50.021394 4823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 03:20:50.024806 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:50.026276 4823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:50.026458 4823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "ca156326fa0f4ea6912079dcf350a7f1"
format_stamp: "Formatted at 2026-09-24 03:20:50 on dist-test-slave-vd0p"
server_key: "af72c00b03d2f4abfa5a2fadfd79d4c1"
server_key_iv: "cddbc00c2a87cdfa660ca9cd9b189b38"
server_key_version: "encryptionkey@0"
I20260924 03:20:50.026648 4823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:50.064643 4823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:50.070775 4823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:50.071127 4823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:50.090071 4823 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:40501
I20260924 03:20:50.090103 4889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:40501 every 8 connection(s)
I20260924 03:20:50.091426 4823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:50.091957 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4823
I20260924 03:20:50.092135 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:50.092780 15246 external_mini_cluster.cc:1428] Setting key 8558ea2129f8de81d0700587d753feeb
I20260924 03:20:50.095719 4890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 03:20:50 dist-test-slave-vd0p krb5kdc[4810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:50.108482 4890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1: Bootstrap starting.
I20260924 03:20:50.111537 4890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:50.112584 4890 log.cc:824] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:50.115852 4890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1: No bootstrap required, opened a new log
I20260924 03:20:50.116294 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:50.094625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37759 (local address 127.14.227.190:40501)
0924 03:20:50.095440 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:50.095452 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:50.095493 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:20:50.097154 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:50.097176 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:50.097545 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:50.097851 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:50.099664 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:50.100983 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:50.102811 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:50.103481 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:50.103614 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:50.108323 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:50.108334 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:50.108353 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:50.108390 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:50.111772 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:50.112512 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:50.112520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:50.112525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:50.112762 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:50.113282 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:50.113285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:50.113291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:50.113845 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:50.114206 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:50.114583 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 03:20:50.115021 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":381,"threads_started":1}
I20260924 03:20:50.120343 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:50.124130 4890 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca156326fa0f4ea6912079dcf350a7f1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40501 } }
I20260924 03:20:50.124478 4890 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:50.124604 4890 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca156326fa0f4ea6912079dcf350a7f1, State: Initialized, Role: FOLLOWER
I20260924 03:20:50.125217 4890 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca156326fa0f4ea6912079dcf350a7f1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40501 } }
I20260924 03:20:50.125501 4890 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:50.125656 4890 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:50.125844 4890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:50.128324 4890 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca156326fa0f4ea6912079dcf350a7f1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40501 } }
I20260924 03:20:50.128880 4890 leader_election.cc:302] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca156326fa0f4ea6912079dcf350a7f1; no voters:
I20260924 03:20:50.129597 4890 leader_election.cc:288] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:50.129688 4900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:50.132133 4890 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:50.132495 4900 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [term 1 LEADER]: Becoming Leader. State: Replica: ca156326fa0f4ea6912079dcf350a7f1, State: Running, Role: LEADER
I20260924 03:20:50.133176 4900 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca156326fa0f4ea6912079dcf350a7f1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40501 } }
I20260924 03:20:50.136014 4901 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca156326fa0f4ea6912079dcf350a7f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca156326fa0f4ea6912079dcf350a7f1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40501 } } }
I20260924 03:20:50.136387 4901 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:50.136853 4902 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca156326fa0f4ea6912079dcf350a7f1. Latest consensus state: current_term: 1 leader_uuid: "ca156326fa0f4ea6912079dcf350a7f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca156326fa0f4ea6912079dcf350a7f1" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40501 } } }
I20260924 03:20:50.138789 4902 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1 [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:50.143278 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:50.125100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38383 (local address 127.14.227.190:40501)
0924 03:20:50.125294 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:50.125300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:50.125826 (+ 526us) server_negotiation.cc:407] Connection header received
0924 03:20:50.126075 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:50.126081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:50.126169 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:50.126273 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:50.127784 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:50.128858 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:50.130720 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:50.131250 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:50.131372 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:50.134531 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:50.134540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:50.134548 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:50.134594 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:50.138944 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:50.139757 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:50.139763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:50.139766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:50.139949 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:50.140379 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:50.140383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:50.140387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:50.140970 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:50.141304 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:50.142471 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0924 03:20:50.142755 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 03:20:50.146974 4905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:50.151893 4905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:50.160617 4905 catalog_manager.cc:1347] Generated new cluster ID: 42684a603557466d913901537c0e4c96
I20260924 03:20:50.160725 4905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:50.176373 4905 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:50.178248 4905 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:50.184952 4905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca156326fa0f4ea6912079dcf350a7f1: Generated new TSK 0
I20260924 03:20:50.185930 4905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:50.192965 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4823
I20260924 03:20:50.194104 4885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:50.282027 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4823
2026-09-24T03:20:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:50 dist-test-slave-vd0p krb5kdc[4922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:50 dist-test-slave-vd0p krb5kdc[4922](info): set up 2 sockets
Sep 24 03:20:50 dist-test-slave-vd0p krb5kdc[4922](info): commencing operation
krb5kdc: starting...
W20260924 03:20:51.374416 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:51 dist-test-slave-vd0p krb5kdc[4922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:51.633985 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:35261
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:51.960520 4935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:51.960822 4935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:51.961022 4935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:51.968189 4935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:51.968293 4935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:51.968382 4935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:51.968472 4935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:51.968554 4935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:51.978355 4935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:35261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:51.979681 4935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:51.981032 4935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:51.993046 4940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:51.993103 4943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:51.993050 4941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:51.993973 4935 server_base.cc:1034] running on GCE node
I20260924 03:20:51.994757 4935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:51.996148 4935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:51.997372 4935 hybrid_clock.cc:648] HybridClock initialized: now 1790220051997336 us; error 51 us; skew 500 ppm
I20260924 03:20:51.997725 4935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:51 dist-test-slave-vd0p krb5kdc[4922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220051, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:52.000844 4935 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:52.002368 4935 webserver.cc:460] Webserver started at http://127.14.227.190:46535/ using document root <none> and password file <none>
I20260924 03:20:52.003048 4935 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:52.003190 4935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:52.003479 4935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:52.006417 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "fea125961dba40629e194d0d504481ba"
format_stamp: "Formatted at 2026-09-24 03:20:52 on dist-test-slave-vd0p"
server_key: "09ebe78c8487a853d8ab41a4fedced13"
server_key_iv: "8329f6997efe9851429764acbd441026"
server_key_version: "encryptionkey@0"
I20260924 03:20:52.007228 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "fea125961dba40629e194d0d504481ba"
format_stamp: "Formatted at 2026-09-24 03:20:52 on dist-test-slave-vd0p"
server_key: "09ebe78c8487a853d8ab41a4fedced13"
server_key_iv: "8329f6997efe9851429764acbd441026"
server_key_version: "encryptionkey@0"
I20260924 03:20:52.012379 4935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:20:52.015920 4950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:52.017359 4935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 03:20:52.017581 4935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "fea125961dba40629e194d0d504481ba"
format_stamp: "Formatted at 2026-09-24 03:20:52 on dist-test-slave-vd0p"
server_key: "09ebe78c8487a853d8ab41a4fedced13"
server_key_iv: "8329f6997efe9851429764acbd441026"
server_key_version: "encryptionkey@0"
I20260924 03:20:52.017787 4935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:52.059126 4935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:52.063740 4935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:52.064090 4935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:52.083770 4935 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:35261
I20260924 03:20:52.083801 5001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:35261 every 8 connection(s)
I20260924 03:20:52.085134 4935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:52.088760 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 4935
I20260924 03:20:52.088842 5002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:52.088966 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:52.089413 15246 external_mini_cluster.cc:1428] Setting key 23c1cda6aead8279f2816b8ed4f6c739
I20260924 03:20:52.102334 5002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba: Bootstrap starting.
I20260924 03:20:52.102687 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:52.091212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53369 (local address 127.14.227.190:35261)
0924 03:20:52.092085 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:52.092096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:52.092135 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:20:52.093673 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:52.093694 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:52.094141 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:52.094473 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:52.095067 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:52.096404 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:52.098169 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:52.098893 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:52.099020 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:52.100247 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:52.100253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:52.100269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:52.100806 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:52.100877 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:52.100901 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:52.100985 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:52.101269 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 03:20:52.101583 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":415,"threads_started":1}
I20260924 03:20:52.105465 5002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:52.106312 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:52.106724 5002 log.cc:824] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:52.109460 5002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba: No bootstrap required, opened a new log
I20260924 03:20:52.117115 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:52.110996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37537 (local address 127.14.227.190:35261)
0924 03:20:52.111229 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:52.111239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:52.111764 (+ 525us) server_negotiation.cc:407] Connection header received
0924 03:20:52.111974 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:52.111978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:52.112047 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:52.112153 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:52.112681 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:52.113401 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:52.114757 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:52.115215 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:52.115356 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:52.116414 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:52.116419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:52.116422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:52.116438 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:52.116490 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:52.116505 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:52.116599 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:52.116741 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 03:20:52.116830 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:20:52.116832 5002 raft_consensus.cc:348] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea125961dba40629e194d0d504481ba" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 35261 } }
I20260924 03:20:52.117383 5002 raft_consensus.cc:374] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:52.117535 5002 raft_consensus.cc:729] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea125961dba40629e194d0d504481ba, State: Initialized, Role: FOLLOWER
I20260924 03:20:52.118214 5002 consensus_queue.cc:260] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea125961dba40629e194d0d504481ba" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 35261 } }
I20260924 03:20:52.118541 5002 raft_consensus.cc:388] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:52.118697 5002 raft_consensus.cc:482] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:52.118851 5002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:52.120882 5002 raft_consensus.cc:504] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea125961dba40629e194d0d504481ba" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 35261 } }
I20260924 03:20:52.121377 5002 leader_election.cc:302] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fea125961dba40629e194d0d504481ba; no voters:
I20260924 03:20:52.122035 5002 leader_election.cc:288] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:52.122170 5012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:52.122893 5012 raft_consensus.cc:686] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [term 1 LEADER]: Becoming Leader. State: Replica: fea125961dba40629e194d0d504481ba, State: Running, Role: LEADER
I20260924 03:20:52.123517 5012 consensus_queue.cc:237] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea125961dba40629e194d0d504481ba" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 35261 } }
I20260924 03:20:52.124194 5002 sys_catalog.cc:564] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:52.126353 5014 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader fea125961dba40629e194d0d504481ba. Latest consensus state: current_term: 1 leader_uuid: "fea125961dba40629e194d0d504481ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea125961dba40629e194d0d504481ba" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 35261 } } }
I20260924 03:20:52.126739 5014 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:52.127168 5013 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fea125961dba40629e194d0d504481ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea125961dba40629e194d0d504481ba" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 35261 } } }
I20260924 03:20:52.127453 5013 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:52.130162 5018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:52.135257 5018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:52.143873 5018 catalog_manager.cc:1347] Generated new cluster ID: f08b8b26efeb4d57bf79f212c6f610e4
I20260924 03:20:52.144004 5018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:52.162724 5018 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:52.164081 5018 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:52.172987 5018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fea125961dba40629e194d0d504481ba: Generated new TSK 0
I20260924 03:20:52.173799 5018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:52.179668 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4935
W20260924 03:20:52.180703 4997 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53369
W20260924 03:20:52.181361 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 4935: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:52.181483 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 4935
2026-09-24T03:20:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:52 dist-test-slave-vd0p krb5kdc[5033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:52 dist-test-slave-vd0p krb5kdc[5033](info): set up 2 sockets
Sep 24 03:20:52 dist-test-slave-vd0p krb5kdc[5033](info): commencing operation
krb5kdc: starting...
W20260924 03:20:53.278740 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:53 dist-test-slave-vd0p krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:53.536755 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:34821
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:34821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:53.857491 5046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:53.857779 5046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:53.857985 5046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:53.865021 5046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:53.865118 5046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:53.865228 5046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:53.865342 5046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:53.865420 5046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:53.874956 5046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:34821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:34821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:53.876374 5046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:53.877646 5046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:53.889595 5054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:53.889561 5051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:53.889644 5052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:53.890568 5046 server_base.cc:1034] running on GCE node
I20260924 03:20:53.891290 5046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:53.892625 5046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:53.893862 5046 hybrid_clock.cc:648] HybridClock initialized: now 1790220053893839 us; error 53 us; skew 500 ppm
I20260924 03:20:53.894217 5046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:53 dist-test-slave-vd0p krb5kdc[5033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220053, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:53.902740 5046 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:53.904294 5046 webserver.cc:460] Webserver started at http://127.14.227.190:41449/ using document root <none> and password file <none>
I20260924 03:20:53.904974 5046 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:53.905121 5046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:53.905427 5046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:53.908385 5046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "f30089ec4f0348d29a69c81101ad276f"
format_stamp: "Formatted at 2026-09-24 03:20:53 on dist-test-slave-vd0p"
server_key: "81e90c6aac833853249f74187d31e337"
server_key_iv: "651961fd9d5221cda9d350764e9c6da6"
server_key_version: "encryptionkey@0"
I20260924 03:20:53.909144 5046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "f30089ec4f0348d29a69c81101ad276f"
format_stamp: "Formatted at 2026-09-24 03:20:53 on dist-test-slave-vd0p"
server_key: "81e90c6aac833853249f74187d31e337"
server_key_iv: "651961fd9d5221cda9d350764e9c6da6"
server_key_version: "encryptionkey@0"
I20260924 03:20:53.914216 5046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 03:20:53.917514 5061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:53.918843 5046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:53.919021 5046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "f30089ec4f0348d29a69c81101ad276f"
format_stamp: "Formatted at 2026-09-24 03:20:53 on dist-test-slave-vd0p"
server_key: "81e90c6aac833853249f74187d31e337"
server_key_iv: "651961fd9d5221cda9d350764e9c6da6"
server_key_version: "encryptionkey@0"
I20260924 03:20:53.919222 5046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:53.924775 5046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:53.928478 5046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:53.928771 5046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:53.946686 5046 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:34821
I20260924 03:20:53.946720 5112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:34821 every 8 connection(s)
I20260924 03:20:53.948007 5046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:53.950788 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 5046
I20260924 03:20:53.950955 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:53.951464 15246 external_mini_cluster.cc:1428] Setting key abc3264086a912790eb55e32571bc91d
I20260924 03:20:53.952119 5113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:53.959565 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:53.953208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54533 (local address 127.14.227.190:34821)
0924 03:20:53.954034 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:53.954050 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:53.954088 (+ 38us) server_negotiation.cc:407] Connection header received
0924 03:20:53.956420 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:53.956440 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:53.956761 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:53.957101 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:53.958103 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:53.958112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:53.958124 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:53.958140 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:53.958200 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:53.958221 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:53.958389 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:53.958487 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 03:20:53.958677 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":364,"threads_started":1}
I20260924 03:20:53.963011 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:53.964643 5113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f: Bootstrap starting.
I20260924 03:20:53.968077 5113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:53.969234 5113 log.cc:824] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:53.969561 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:53.967072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59121 (local address 127.14.227.190:34821)
0924 03:20:53.967212 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:53.967216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:53.967689 (+ 473us) server_negotiation.cc:407] Connection header received
0924 03:20:53.967874 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:53.967879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:53.967933 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:53.968039 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:53.968950 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:53.968955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:53.968959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:20:53.968985 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:53.969020 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:20:53.969034 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:53.969155 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:53.969252 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 03:20:53.969308 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:20:53.972095 5113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f: No bootstrap required, opened a new log
I20260924 03:20:53.979749 5113 raft_consensus.cc:348] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30089ec4f0348d29a69c81101ad276f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34821 } }
I20260924 03:20:53.980118 5113 raft_consensus.cc:374] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:20:53.980218 5113 raft_consensus.cc:729] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30089ec4f0348d29a69c81101ad276f, State: Initialized, Role: FOLLOWER
I20260924 03:20:53.980881 5113 consensus_queue.cc:260] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30089ec4f0348d29a69c81101ad276f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34821 } }
I20260924 03:20:53.981153 5113 raft_consensus.cc:388] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:20:53.981288 5113 raft_consensus.cc:482] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:20:53.981472 5113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:53.983672 5113 raft_consensus.cc:504] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30089ec4f0348d29a69c81101ad276f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34821 } }
I20260924 03:20:53.984154 5113 leader_election.cc:302] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f30089ec4f0348d29a69c81101ad276f; no voters:
I20260924 03:20:53.984717 5113 leader_election.cc:288] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:20:53.984877 5123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:53.985638 5123 raft_consensus.cc:686] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [term 1 LEADER]: Becoming Leader. State: Replica: f30089ec4f0348d29a69c81101ad276f, State: Running, Role: LEADER
I20260924 03:20:53.986246 5123 consensus_queue.cc:237] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30089ec4f0348d29a69c81101ad276f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34821 } }
I20260924 03:20:53.986967 5113 sys_catalog.cc:564] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:53.989275 5124 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f30089ec4f0348d29a69c81101ad276f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30089ec4f0348d29a69c81101ad276f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34821 } } }
I20260924 03:20:53.989825 5124 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:53.989303 5125 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f30089ec4f0348d29a69c81101ad276f. Latest consensus state: current_term: 1 leader_uuid: "f30089ec4f0348d29a69c81101ad276f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30089ec4f0348d29a69c81101ad276f" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 34821 } } }
I20260924 03:20:53.990266 5125 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f [sys.catalog]: This master's current role is: LEADER
W20260924 03:20:53.998710 5139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:20:53.998811 5139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:20:53.998939 5128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:54.001147 5128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:20:54.009559 5128 catalog_manager.cc:1347] Generated new cluster ID: b6ad6c7fbf5c40139fcf9cdaad46847a
I20260924 03:20:54.009666 5128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:54.068111 5128 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:54.068702 5128 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:54.084242 5128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f30089ec4f0348d29a69c81101ad276f: Generated new TSK 0
I20260924 03:20:54.085119 5128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:54.105665 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 5046
W20260924 03:20:54.107111 5108 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54533
W20260924 03:20:54.107949 15246 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 5046: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 03:20:54.108121 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 5046
2026-09-24T03:20:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1928 ms)
[----------] 18 tests from AuthTokenIssuingTest (34501 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:54 dist-test-slave-vd0p krb5kdc[5144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:54 dist-test-slave-vd0p krb5kdc[5144](info): set up 2 sockets
Sep 24 03:20:54 dist-test-slave-vd0p krb5kdc[5144](info): commencing operation
krb5kdc: starting...
W20260924 03:20:55.206700 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:55 dist-test-slave-vd0p krb5kdc[5144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:55.476945 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:42145
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:42145,127.14.227.189:45373,127.14.227.188:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:20:55.846840 5157 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:20:55.847211 5157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:55.847334 5157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:55.847478 5157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:55.854846 5157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:55.854936 5157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:55.855051 5157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:55.855115 5157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:55.855250 5157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:55.865314 5157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:42145,127.14.227.189:45373,127.14.227.188:41505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:42145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:55.866626 5157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:55.868496 5157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:55.882896 5163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:55.882771 5162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:55.882872 5165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:55.883836 5157 server_base.cc:1034] running on GCE node
I20260924 03:20:55.884619 5157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:55.886042 5157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:55.887277 5157 hybrid_clock.cc:648] HybridClock initialized: now 1790220055887203 us; error 53 us; skew 500 ppm
I20260924 03:20:55.887611 5157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:55 dist-test-slave-vd0p krb5kdc[5144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220055, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:55.890753 5157 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:20:55.892237 5157 webserver.cc:460] Webserver started at http://127.14.227.190:33339/ using document root <none> and password file <none>
I20260924 03:20:55.893014 5157 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:55.893118 5157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:55.893457 5157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:55.896427 5157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "19073e27fc5f480b84ad5c8434d3a44a"
format_stamp: "Formatted at 2026-09-24 03:20:55 on dist-test-slave-vd0p"
server_key: "47ba9b412ec407f3c1711904071a3167"
server_key_iv: "75bd1afb874817247e10a681f39e4dbf"
server_key_version: "encryptionkey@0"
I20260924 03:20:55.897230 5157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "19073e27fc5f480b84ad5c8434d3a44a"
format_stamp: "Formatted at 2026-09-24 03:20:55 on dist-test-slave-vd0p"
server_key: "47ba9b412ec407f3c1711904071a3167"
server_key_iv: "75bd1afb874817247e10a681f39e4dbf"
server_key_version: "encryptionkey@0"
I20260924 03:20:55.902282 5157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 03:20:55.905812 5172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:55.907173 5157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:55.907348 5157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "19073e27fc5f480b84ad5c8434d3a44a"
format_stamp: "Formatted at 2026-09-24 03:20:55 on dist-test-slave-vd0p"
server_key: "47ba9b412ec407f3c1711904071a3167"
server_key_iv: "75bd1afb874817247e10a681f39e4dbf"
server_key_version: "encryptionkey@0"
I20260924 03:20:55.907544 5157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:55.924489 5157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:55.928404 5157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:55.928627 5157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:55.946409 5157 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:42145
I20260924 03:20:55.946514 5223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:42145 every 8 connection(s)
I20260924 03:20:55.947755 5157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:20:55.951709 5224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:55.953795 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 5157
I20260924 03:20:55.954056 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:20:55.954697 15246 external_mini_cluster.cc:1428] Setting key 6d90b16b04ee2dd9eb5b332e2d301b4d
I20260924 03:20:55.958590 5224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:20:55 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220055, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.227.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.189@KRBTEST.COM" created.
I20260924 03:20:55.983639 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:55.961822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:42145 (local address 127.0.0.1:41784)
0924 03:20:55.962855 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:55.962924 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:55.964776 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:55.965752 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:55.965757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:55.966389 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:55.966726 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:55.966746 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:55.969113 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:55.969117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:55.970247 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:55.970258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:55.971653 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:55.971657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:55.971770 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:55.972463 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:55.972483 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:55.974579 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:55.979273 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:55.979282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:55.979291 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:55.979676 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:55.980101 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:55.980108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:55.980111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:55.980247 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:55.981140 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:55.981148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:55.981349 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 03:20:55.981645 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 03:20:55.982265 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":156,"thread_start_us":433,"threads_started":1}
I20260924 03:20:55.983675 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:55.961802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.14.227.190:42145)
0924 03:20:55.963610 (+ 1808us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:55.963621 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:55.963686 (+ 65us) server_negotiation.cc:407] Connection header received
0924 03:20:55.965150 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:55.965161 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:55.965498 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:55.965742 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:55.966957 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:55.968937 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:55.970388 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:55.971446 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:55.971654 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:55.974918 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:55.974931 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:55.974950 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:55.975000 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:55.979135 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:55.979814 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:55.979827 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:55.979829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:55.979992 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:55.980357 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:55.980368 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:55.980371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:55.980994 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:55.981195 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:55.981738 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 03:20:55.982155 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1558,"thread_start_us":1348,"threads_started":1}
I20260924 03:20:55.986457 5224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:20:55.987555 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:55.987168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38286)
0924 03:20:55.987337 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:55.987447 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 03:20:55.987816 5174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
I20260924 03:20:55.989310 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:55.988942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38290)
0924 03:20:55.989102 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:55.989162 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 03:20:55.989601 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:45373: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab.
Entry for principal kudu/127.14.227.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab.
I20260924 03:20:56.010912 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.189:45373
--webserver_interface=127.14.227.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab
--principal=kudu/127.14.227.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:42145,127.14.227.189:45373,127.14.227.188:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:20:56.038932 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } attempt: 1
I20260924 03:20:56.040161 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.039686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38298)
0924 03:20:56.039873 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.040028 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:20:56.041113 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.040802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38314)
0924 03:20:56.040923 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.040993 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 03:20:56.041322 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:45373: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
I20260924 03:20:56.109573 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } attempt: 2
I20260924 03:20:56.110888 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.110357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38320)
0924 03:20:56.110602 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.110695 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 03:20:56.111883 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.111528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38332)
0924 03:20:56.111685 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.111752 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:20:56.112138 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:45373: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
I20260924 03:20:56.203369 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } attempt: 3
I20260924 03:20:56.204586 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.204152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38342)
0924 03:20:56.204348 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.204438 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 03:20:56.205575 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.205213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38358)
0924 03:20:56.205357 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.205434 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 03:20:56.205736 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:45373: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
W20260924 03:20:56.335301 5231 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:20:56.335680 5231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:56.335811 5231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:56.335970 5231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:56.343219 5231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:56.343343 5231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:56.343477 5231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:56.343561 5231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:56.343652 5231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:56.349076 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } attempt: 4
I20260924 03:20:56.350320 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.349691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38368)
0924 03:20:56.350049 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.350155 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 03:20:56.351336 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.350954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38382)
0924 03:20:56.351101 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.351194 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 03:20:56.351581 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:45373: Network error: Client connection negotiation failed: client connection to 127.14.227.189:45373: connect: Connection refused (error 111)
I20260924 03:20:56.354322 5231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:42145,127.14.227.189:45373,127.14.227.188:41505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab
--principal=kudu/127.14.227.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.189:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.227.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:56.355511 5231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:56.356726 5231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:20:56.368213 5236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:56.368189 5237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:56.368657 5239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:56.368963 5231 server_base.cc:1034] running on GCE node
I20260924 03:20:56.369755 5231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:56.371191 5231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:56.372438 5231 hybrid_clock.cc:648] HybridClock initialized: now 1790220056372358 us; error 57 us; skew 500 ppm
I20260924 03:20:56.372766 5231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220056, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:56.375845 5231 init.cc:377] Logged in from keytab as kudu/127.14.227.189@KRBTEST.COM (short username kudu)
I20260924 03:20:56.377327 5231 webserver.cc:460] Webserver started at http://127.14.227.189:35863/ using document root <none> and password file <none>
I20260924 03:20:56.378082 5231 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:56.378208 5231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:56.378539 5231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:56.381474 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/data/instance:
uuid: "c52d1b7886854a578af6b7c6819abf0e"
format_stamp: "Formatted at 2026-09-24 03:20:56 on dist-test-slave-vd0p"
server_key: "f12f1835a220a30d00c4d95278457165"
server_key_iv: "eb1ded5b7ef0ee6afe21923e264c84f6"
server_key_version: "encryptionkey@0"
I20260924 03:20:56.382328 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/wal/instance:
uuid: "c52d1b7886854a578af6b7c6819abf0e"
format_stamp: "Formatted at 2026-09-24 03:20:56 on dist-test-slave-vd0p"
server_key: "f12f1835a220a30d00c4d95278457165"
server_key_iv: "eb1ded5b7ef0ee6afe21923e264c84f6"
server_key_version: "encryptionkey@0"
I20260924 03:20:56.387243 5231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:20:56.390661 5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:56.392055 5231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:20:56.392186 5231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/wal
uuid: "c52d1b7886854a578af6b7c6819abf0e"
format_stamp: "Formatted at 2026-09-24 03:20:56 on dist-test-slave-vd0p"
server_key: "f12f1835a220a30d00c4d95278457165"
server_key_iv: "eb1ded5b7ef0ee6afe21923e264c84f6"
server_key_version: "encryptionkey@0"
I20260924 03:20:56.392352 5231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:56.408197 5231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:56.412245 5231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:56.412463 5231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:56.431087 5231 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.189:45373
I20260924 03:20:56.431110 5297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.189:45373 every 8 connection(s)
I20260924 03:20:56.432456 5231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
I20260924 03:20:56.433925 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 5231
I20260924 03:20:56.434151 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-1/wal/instance
I20260924 03:20:56.434686 15246 external_mini_cluster.cc:1428] Setting key db05321f880a89272aeef378526f5b4f
I20260924 03:20:56.436978 5298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:56.447065 5298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.227.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.188@KRBTEST.COM" created.
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220056, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:56.471650 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.449670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.14.227.190:42145)
0924 03:20:56.449902 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.449911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.451187 (+ 1276us) server_negotiation.cc:407] Connection header received
0924 03:20:56.453051 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.453060 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.453171 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.453297 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.454872 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.456108 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.457765 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.458610 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.458805 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.464403 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.464413 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.464420 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.464468 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.468239 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.468993 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.469001 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.469007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.469155 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.469580 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.469587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.469593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.470024 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.470179 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.470941 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.471203 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:20:56.472499 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.450186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:42145 (local address 127.0.0.1:41790)
0924 03:20:56.451015 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.451059 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:56.452784 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:56.453514 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:56.453522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:56.454118 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:56.454697 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.454714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.456271 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.456275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.457591 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.457602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.458845 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.458849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.459058 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.459738 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:56.459761 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:56.464147 (+ 4386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:56.468405 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.468412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.468426 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.468849 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.469261 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.469264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.469266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.469457 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.470165 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:56.470172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:56.470548 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 03:20:56.470788 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 03:20:56.471189 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":377,"threads_started":1}
I20260924 03:20:56.474370 5298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220056, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:20:56.488751 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.475212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38388)
0924 03:20:56.475409 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.475436 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:56.475555 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:56.476308 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:56.476311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:56.476678 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:56.476815 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.476823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.478246 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.478254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.479119 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.479129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.479935 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.479940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.480136 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.480824 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:56.480842 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:56.483140 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:56.486316 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.486324 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.486329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.486674 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.487035 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.487042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.487047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.487188 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.487780 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:56.487787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:56.487927 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 03:20:56.488065 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 03:20:56.488336 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 03:20:56.488795 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.475224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38388 (local address 127.14.227.189:45373)
0924 03:20:56.475834 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.475844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.475874 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:20:56.475932 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.475943 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.476168 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.476313 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.477002 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.478128 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.479249 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.479795 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.479927 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.483341 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.483351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.483365 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.483392 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.486199 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.486797 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.486811 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.486815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.486915 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.487296 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.487322 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.487325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.487657 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.487861 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.488225 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.488433 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":268,"threads_started":1}
I20260924 03:20:56.491475 5298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:20:56.492753 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.492223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60894)
0924 03:20:56.492462 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.492601 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 03:20:56.493057 5247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab.
Entry for principal kudu/127.14.227.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab.
I20260924 03:20:56.494673 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.494214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60896)
0924 03:20:56.494423 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.494504 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 03:20:56.494915 5298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
I20260924 03:20:56.495695 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.188:41505
--webserver_interface=127.14.227.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab
--principal=kudu/127.14.227.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:42145,127.14.227.189:45373,127.14.227.188:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:20:56.544284 5298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 1
I20260924 03:20:56.545532 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.545076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60912)
0924 03:20:56.545266 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.545383 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 03:20:56.546850 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.546464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60914)
0924 03:20:56.546653 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.546736 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 03:20:56.547214 5298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
I20260924 03:20:56.615455 5298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 2
I20260924 03:20:56.616768 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.616287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60928)
0924 03:20:56.616522 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.616615 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 03:20:56.617883 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.617434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60944)
0924 03:20:56.617622 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.617725 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 03:20:56.618183 5298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
I20260924 03:20:56.650837 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } attempt: 5
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220055, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:20:56.664116 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.651650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38404)
0924 03:20:56.651848 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.651872 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:56.652020 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:56.652455 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:56.652461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:56.652854 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:56.653047 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.653055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.654095 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.654101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.654950 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.654961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.655732 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.655738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.655948 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.656571 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:56.656587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:56.658725 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:56.661647 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.661652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.661656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.662105 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.662470 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.662475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.662478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.662634 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.663144 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:56.663148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:56.663299 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 03:20:56.663444 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 03:20:56.663750 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 03:20:56.664113 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.651789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.14.227.189:45373)
0924 03:20:56.651984 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.651990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.652012 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:20:56.652216 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.652222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.652308 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.652413 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.653204 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.653933 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.655155 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.655591 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.655726 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.658907 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.658911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.658914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.658945 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.661415 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.662220 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.662231 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.662233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.662360 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.662739 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.662745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.662748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.663027 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.663179 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.663545 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.663765 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 03:20:56.665282 5224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:20:56.666477 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.666120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60952)
0924 03:20:56.666281 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.666354 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 03:20:56.667455 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.667137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60962)
0924 03:20:56.667282 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.667355 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:20:56.667683 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
I20260924 03:20:56.709434 5298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 3
I20260924 03:20:56.710675 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.710209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60972)
0924 03:20:56.710419 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.710520 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 03:20:56.711840 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.711440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60980)
0924 03:20:56.711656 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.711735 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260924 03:20:56.712042 5298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
I20260924 03:20:56.718873 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 1
I20260924 03:20:56.719914 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.719433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:60994)
0924 03:20:56.719631 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.719731 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 03:20:56.720925 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.720578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32776)
0924 03:20:56.720720 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.720801 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 03:20:56.721168 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
I20260924 03:20:56.789398 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 2
I20260924 03:20:56.790706 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.790079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32788)
0924 03:20:56.790348 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.790477 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 03:20:56.791855 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.791429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32796)
0924 03:20:56.791603 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.791697 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:20:56.792074 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
W20260924 03:20:56.825033 5305 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:20:56.825350 5305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:20:56.825457 5305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:20:56.825601 5305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:20:56.832996 5305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:20:56.833086 5305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:20:56.833213 5305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:20:56.833292 5305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:20:56.833405 5305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:20:56.843119 5305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:42145,127.14.227.189:45373,127.14.227.188:41505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab
--principal=kudu/127.14.227.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.188:41505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.227.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:20:56.844388 5305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:20:56.845716 5305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:20:56.855257 5298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 4
I20260924 03:20:56.856638 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.855907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32808)
0924 03:20:56.856312 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.856428 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260924 03:20:56.857306 5311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:20:56.857718 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.857350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32812)
0924 03:20:56.857508 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.857592 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 03:20:56.857725 5313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:56.857836 5310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:20:56.857975 5298 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
I20260924 03:20:56.858609 5305 server_base.cc:1034] running on GCE node
I20260924 03:20:56.859525 5305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:20:56.860987 5305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:20:56.862272 5305 hybrid_clock.cc:648] HybridClock initialized: now 1790220056862187 us; error 55 us; skew 500 ppm
I20260924 03:20:56.862660 5305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220056, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:20:56.865790 5305 init.cc:377] Logged in from keytab as kudu/127.14.227.188@KRBTEST.COM (short username kudu)
I20260924 03:20:56.867316 5305 webserver.cc:460] Webserver started at http://127.14.227.188:34533/ using document root <none> and password file <none>
I20260924 03:20:56.868038 5305 fs_manager.cc:359] Metadata directory not provided
I20260924 03:20:56.868176 5305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:20:56.868497 5305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:20:56.871526 5305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/data/instance:
uuid: "9e369622cb3b4b55884a1d64c9f817dd"
format_stamp: "Formatted at 2026-09-24 03:20:56 on dist-test-slave-vd0p"
server_key: "3e3b987b905320dfebfef40af1242527"
server_key_iv: "30fbb3ef0bec619a626fb6d63248473b"
server_key_version: "encryptionkey@0"
I20260924 03:20:56.872303 5305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/wal/instance:
uuid: "9e369622cb3b4b55884a1d64c9f817dd"
format_stamp: "Formatted at 2026-09-24 03:20:56 on dist-test-slave-vd0p"
server_key: "3e3b987b905320dfebfef40af1242527"
server_key_iv: "30fbb3ef0bec619a626fb6d63248473b"
server_key_version: "encryptionkey@0"
I20260924 03:20:56.877316 5305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 03:20:56.880690 5320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:20:56.882109 5305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 03:20:56.882293 5305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/wal
uuid: "9e369622cb3b4b55884a1d64c9f817dd"
format_stamp: "Formatted at 2026-09-24 03:20:56 on dist-test-slave-vd0p"
server_key: "3e3b987b905320dfebfef40af1242527"
server_key_iv: "30fbb3ef0bec619a626fb6d63248473b"
server_key_version: "encryptionkey@0"
I20260924 03:20:56.882521 5305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:20:56.898315 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 3
I20260924 03:20:56.899452 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.898950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32820)
0924 03:20:56.899145 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.899277 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 03:20:56.900429 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.900094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32826)
0924 03:20:56.900256 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.900328 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 03:20:56.900710 5224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:41505: Network error: Client connection negotiation failed: client connection to 127.14.227.188:41505: connect: Connection refused (error 111)
I20260924 03:20:56.901300 5305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:20:56.905158 5305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:20:56.905416 5305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:20:56.923908 5305 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.188:41505
I20260924 03:20:56.923925 5371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.188:41505 every 8 connection(s)
I20260924 03:20:56.925298 5305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
I20260924 03:20:56.929042 5372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:20:56.930104 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 5305
I20260924 03:20:56.930294 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790219707325830-15246-0/minicluster-data/master-2/wal/instance
I20260924 03:20:56.930723 15246 external_mini_cluster.cc:1428] Setting key 1411b251ba790af5c1d4de20db0e0f0d
I20260924 03:20:56.938328 5372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220056, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:56.955464 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.932607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.14.227.188:41505)
0924 03:20:56.933397 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.933412 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.933455 (+ 43us) server_negotiation.cc:407] Connection header received
0924 03:20:56.935109 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.935132 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.935547 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.935883 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.937686 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.939089 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.940815 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.942102 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.942275 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.946454 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.946463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.946477 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.946517 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.950271 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.951084 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.951092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.951100 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.951277 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.951727 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.951733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.951736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.952453 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.953041 (+ 588us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.953589 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.954049 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":373,"threads_started":1}
I20260924 03:20:56.957290 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.941252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.14.227.190:42145)
0924 03:20:56.941421 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.941426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.942521 (+ 1095us) server_negotiation.cc:407] Connection header received
0924 03:20:56.942714 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.942719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.942780 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.942898 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.943991 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.944656 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.946335 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.946825 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.946971 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.950724 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.950732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.950738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.950790 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.954306 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.955143 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.955148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.955152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.955319 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.955784 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.955789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.955792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.956066 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.956169 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.956697 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.956908 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 03:20:56.957329 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.941454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:42145 (local address 127.0.0.1:41794)
0924 03:20:56.942359 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.942412 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:56.942567 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:56.942991 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:56.942998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:56.943600 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:56.943823 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.943831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.944811 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.944816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.946164 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.946177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.947008 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.947015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.947472 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.948105 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:56.948122 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:56.950484 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:56.954456 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.954466 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.954473 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.954998 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.955476 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.955484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.955491 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.955657 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.956240 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:56.956245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:56.956393 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 03:20:56.956585 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 03:20:56.956894 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":449,"threads_started":1}
I20260924 03:20:56.959335 5372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220056, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:20:56.972694 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.959923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:45373 (local address 127.0.0.1:38412)
0924 03:20:56.960122 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.960143 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:56.960232 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:56.960617 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:56.960621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:56.961041 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:56.961213 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.961223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.962235 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.962242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.963204 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.963220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.964154 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.964159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.964299 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.964834 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:56.964850 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:56.966918 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:56.970378 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.970385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.970389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.970743 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.971124 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.971130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.971133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.971268 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.971772 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:56.971776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:56.971931 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 03:20:56.972069 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 03:20:56.972340 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:20:56.972728 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.960195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.14.227.189:45373)
0924 03:20:56.960354 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.960358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.960372 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:20:56.960418 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.960422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.960473 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.960564 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.961361 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.962104 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.963383 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.964006 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.964135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.967090 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.967097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.967101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.967138 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.970251 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.970860 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.970867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.970870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.971015 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.971382 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.971389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.971392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.971632 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.971810 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.972153 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.972355 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 03:20:56.972986 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.958474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.14.227.190:42145)
0924 03:20:56.958699 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.958704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.958726 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:20:56.958919 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.958923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.958979 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.959062 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.960365 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.961367 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.962636 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.963068 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.963198 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.967934 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.967939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.967942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.967969 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.970449 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.971076 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.971083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.971086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.971216 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.971588 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.971593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.971597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.971870 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.972143 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.972411 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.972636 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 03:20:56.974179 5372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220056, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
Sep 24 03:20:56 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:20:56.988011 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.974943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.14.227.188:41505)
0924 03:20:56.975114 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.975119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.975289 (+ 170us) server_negotiation.cc:407] Connection header received
0924 03:20:56.975408 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.975412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.975473 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.975572 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.976178 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.977082 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.978494 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.979168 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.979274 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.982455 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.982462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.982468 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.982515 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.985150 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.985995 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.986001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.986007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.986166 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.986713 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.986717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.986719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.987017 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.987176 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.987573 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.987733 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:20:56.988371 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.974959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32840)
0924 03:20:56.975191 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:56.975208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:56.975312 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:56.975596 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:56.975600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:56.975899 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:56.976018 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.976026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.977199 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.977205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.978368 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:56.978380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.979318 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.979324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:56.979552 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.980153 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:56.980169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:56.982277 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:56.985281 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.985284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.985287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.985805 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.986408 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:56.986415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:56.986420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:56.986598 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:56.987158 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:56.987163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:56.987330 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 03:20:56.987502 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 03:20:56.987921 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":80}
I20260924 03:20:56.993287 5372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: Bootstrap starting.
I20260924 03:20:56.994127 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:56.974798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.14.227.189:45373)
0924 03:20:56.974945 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:56.974950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:56.975068 (+ 118us) server_negotiation.cc:407] Connection header received
0924 03:20:56.975222 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:56.975230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:56.975311 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:56.975392 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:56.976757 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.977668 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.979171 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:56.979814 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:56.979912 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:56.984721 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:56.984727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:56.984731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:56.984770 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:56.987897 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.988639 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.988644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.988646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.988823 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:56.989205 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:56.989210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:56.989213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:56.989549 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:56.993552 (+ 4003us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:56.993654 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 03:20:56.993821 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:20:56.997118 5372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:56.998224 5372 log.cc:824] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:56.999014 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:20:57.000883 5372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: No bootstrap required, opened a new log
I20260924 03:20:57.001742 5199 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "19073e27fc5f480b84ad5c8434d3a44a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41796
I20260924 03:20:57.011597 5372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } }
I20260924 03:20:57.011991 5372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e369622cb3b4b55884a1d64c9f817dd, State: Initialized, Role: FOLLOWER
I20260924 03:20:57.012472 5372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } }
I20260924 03:20:57.013335 5378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.013660 5378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:20:57.014577 5372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:20:57.023672 5389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:20:57.023825 5389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:20:57.021488 5305 master_runner.cc:186] Error getting master registration for 127.14.227.190:42145: OK, instance_id { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" instance_seqno: 1790220055941136 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:20:57.029382 5305 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:20:57.077912 5224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 4
Sep 24 03:20:57 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220055, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
I20260924 03:20:57.094642 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.078700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32848)
0924 03:20:57.078883 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:57.078902 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:57.079049 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:57.079431 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:57.079435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:57.079971 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:57.080203 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:57.080213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.081704 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.081711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:57.083250 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:57.083268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.084291 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.084297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:57.084515 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.085181 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:57.085199 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:57.087687 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:57.091829 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:57.091834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:57.091838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:57.092332 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:57.092737 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:57.092741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:57.092745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:57.092896 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:57.093602 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:57.093608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:57.093777 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 03:20:57.093924 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 03:20:57.094220 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 03:20:57.094677 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.078773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.14.227.188:41505)
0924 03:20:57.079003 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.079009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.079032 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:20:57.079185 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.079191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.079271 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.079386 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.080360 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.081553 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.083417 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.084139 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.084272 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.087878 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:57.087885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:57.087890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:57.087929 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:57.091689 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.092457 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.092463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.092467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.092618 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.093007 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.093013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.093017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.093466 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:57.093632 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.094075 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.094301 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:20:57.100553 5224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a: Bootstrap starting.
I20260924 03:20:57.104550 5224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:57.105822 5224 log.cc:824] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:57.108510 5224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a: No bootstrap required, opened a new log
I20260924 03:20:57.113935 5224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } }
I20260924 03:20:57.114338 5224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19073e27fc5f480b84ad5c8434d3a44a, State: Initialized, Role: FOLLOWER
I20260924 03:20:57.114800 5224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } }
I20260924 03:20:57.115845 5392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.116320 5392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:20:57.117009 5224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:57.117138 5199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:20:57.117333 5199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:57.120628 5199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } }
I20260924 03:20:57.123660 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19073e27fc5f480b84ad5c8434d3a44a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e369622cb3b4b55884a1d64c9f817dd"
I20260924 03:20:57.124148 5347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:57.124531 5199 leader_election.cc:288] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [CANDIDATE]: Term 1 election: Requested vote from peers c52d1b7886854a578af6b7c6819abf0e (127.14.227.189:45373), 9e369622cb3b4b55884a1d64c9f817dd (127.14.227.188:41505)
I20260924 03:20:57.127708 5347 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19073e27fc5f480b84ad5c8434d3a44a in term 1.
I20260924 03:20:57.128341 5173 leader_election.cc:302] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19073e27fc5f480b84ad5c8434d3a44a, 9e369622cb3b4b55884a1d64c9f817dd; no voters:
I20260924 03:20:57.128856 5392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:20:57.140098 5392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [term 1 LEADER]: Becoming Leader. State: Replica: 19073e27fc5f480b84ad5c8434d3a44a, State: Running, Role: LEADER
I20260924 03:20:57.140707 5392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } }
I20260924 03:20:57.150532 5404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19073e27fc5f480b84ad5c8434d3a44a. Latest consensus state: current_term: 1 leader_uuid: "19073e27fc5f480b84ad5c8434d3a44a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.151237 5404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:57.152585 5409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:20:57.157196 5298 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } attempt: 5
W20260924 03:20:57.157892 5326 tablet.cc:2369] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:57.168588 5413 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 03:20:57.177039 5409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:20:57 dist-test-slave-vd0p krb5kdc[5144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220056, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
W20260924 03:20:57.181243 5178 tablet.cc:2369] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:20:57.188532 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.153006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.14.227.189:45373)
0924 03:20:57.153192 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.153198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.163281 (+ 10083us) server_negotiation.cc:407] Connection header received
0924 03:20:57.164373 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.164380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.164453 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.164530 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.168952 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.170168 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.172371 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.174388 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.174512 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.180926 (+ 6414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:57.180933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:57.180938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:57.180977 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:57.185212 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.186049 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.186054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.186056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.186213 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.186699 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.186705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.186708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.187066 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:57.187320 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.187894 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.188129 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1344,"server-negotiator.queue_time_us":62}
I20260924 03:20:57.194320 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.159160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:41505 (local address 127.0.0.1:32872)
0924 03:20:57.159608 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 03:20:57.159635 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 03:20:57.159783 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:20:57.167941 (+ 8158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:20:57.167949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:20:57.168603 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:20:57.168872 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:57.168888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.172392 (+ 3504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.172401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:57.173998 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:20:57.174019 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.176055 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.176061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:20:57.176258 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.176940 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:20:57.176963 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:20:57.180258 (+ 3295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:20:57.189964 (+ 9706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:57.189973 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:57.189979 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:57.190588 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:57.191508 (+ 920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:20:57.191516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:20:57.191522 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:20:57.191744 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:20:57.192981 (+ 1237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:20:57.192989 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:20:57.193192 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 03:20:57.193338 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 03:20:57.193731 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":4}
I20260924 03:20:57.196688 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.160045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.14.227.188:41505)
0924 03:20:57.167529 (+ 7484us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.167538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.167564 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:20:57.167632 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.167640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.167731 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.167861 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.169217 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.172100 (+ 2883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.174799 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.175815 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.176651 (+ 836us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.180742 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:57.180777 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:57.180781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:57.180844 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:57.189685 (+ 8841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.190872 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.190880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.190886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.191292 (+ 406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.191967 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.191974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.191980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.192761 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:57.194842 (+ 2081us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.195111 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.195615 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7303,"thread_start_us":7095,"threads_started":1}
I20260924 03:20:57.186523 5157 master_runner.cc:186] Error getting master registration for 127.14.227.189:45373: OK, instance_id { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" instance_seqno: 1790220056425732 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:20:57.197847 5157 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:20:57.203059 5298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e: Bootstrap starting.
I20260924 03:20:57.207866 5347 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [term 1 FOLLOWER]: Refusing update from remote peer 19073e27fc5f480b84ad5c8434d3a44a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:57.208335 5298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e: Neither blocks nor log segments found. Creating new log.
I20260924 03:20:57.209336 5404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 03:20:57.209899 5298 log.cc:824] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e: Log is configured to *not* fsync() on all Append() calls
I20260924 03:20:57.228053 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.152449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.14.227.188:41505)
0924 03:20:57.152637 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.152643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.176931 (+ 24288us) server_negotiation.cc:407] Connection header received
0924 03:20:57.177404 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.177409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.177488 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.177556 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.200168 (+ 22612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.201207 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.210070 (+ 8863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.210776 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.210904 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.218116 (+ 7212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:57.218123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:57.218128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:57.218172 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:57.221715 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.222588 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.222593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.222597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.222796 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.223275 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.223281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.223284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.223661 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:57.223902 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.227417 (+ 3515us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.227662 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:20:57.242187 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.153217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.14.227.190:42145)
0924 03:20:57.153413 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.153421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.178519 (+ 25098us) server_negotiation.cc:407] Connection header received
0924 03:20:57.181429 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.181435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.181522 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.181604 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.202245 (+ 20641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.203384 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.207415 (+ 4031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.208146 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.208250 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.230094 (+ 21844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:20:57.230104 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:20:57.230112 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:20:57.230172 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:20:57.235247 (+ 5075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.239168 (+ 3921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.239173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.239176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.239356 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:20:57.239828 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:20:57.239833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:20:57.239836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:20:57.240245 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:20:57.240529 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.241248 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.241595 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:20:57.244091 5298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e: No bootstrap required, opened a new log
I20260924 03:20:57.246475 5378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19073e27fc5f480b84ad5c8434d3a44a. Latest consensus state: current_term: 1 leader_uuid: "19073e27fc5f480b84ad5c8434d3a44a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.246802 5378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:20:57.249325 5416 mvcc.cc:205] Tried to move back new op lower bound from 7332741354309120000 to 7332741354054078464. Current Snapshot: MvccSnapshot[applied={T|T < 7332741354309120000}]
I20260924 03:20:57.251230 5418 mvcc.cc:205] Tried to move back new op lower bound from 7332741354309120000 to 7332741354054078464. Current Snapshot: MvccSnapshot[applied={T|T < 7332741354309120000}]
I20260924 03:20:57.254539 5378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19073e27fc5f480b84ad5c8434d3a44a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.254822 5378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:20:57.255323 5298 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } }
I20260924 03:20:57.255739 5298 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52d1b7886854a578af6b7c6819abf0e, State: Initialized, Role: FOLLOWER
I20260924 03:20:57.256253 5298 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } }
I20260924 03:20:57.267922 5392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19073e27fc5f480b84ad5c8434d3a44a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.268388 5392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:57.269096 5409 catalog_manager.cc:1347] Generated new cluster ID: 210122c5105f480e93f1f7aa7cae2801
I20260924 03:20:57.269210 5409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:20:57.270519 5422 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.271044 5422 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:20:57.276953 5298 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:20:57.277489 5272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:20:57.281646 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19073e27fc5f480b84ad5c8434d3a44a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c52d1b7886854a578af6b7c6819abf0e"
I20260924 03:20:57.285197 5272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [term 1 FOLLOWER]: Refusing update from remote peer 19073e27fc5f480b84ad5c8434d3a44a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:20:57.288892 5273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19073e27fc5f480b84ad5c8434d3a44a in term 1.
I20260924 03:20:57.293823 5404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:20:57.307871 5409 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:20:57.309975 5409 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:20:57.327322 5409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a: Generated new TSK 0
I20260924 03:20:57.328495 5409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:20:57.335858 5423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19073e27fc5f480b84ad5c8434d3a44a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.336313 5423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19073e27fc5f480b84ad5c8434d3a44a [sys.catalog]: This master's current role is: LEADER
I20260924 03:20:57.337545 5422 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19073e27fc5f480b84ad5c8434d3a44a. Latest consensus state: current_term: 1 leader_uuid: "19073e27fc5f480b84ad5c8434d3a44a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.337883 5422 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:20:57.337632 5431 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19073e27fc5f480b84ad5c8434d3a44a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19073e27fc5f480b84ad5c8434d3a44a" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 42145 } } peers { permanent_uuid: "c52d1b7886854a578af6b7c6819abf0e" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 45373 } } peers { permanent_uuid: "9e369622cb3b4b55884a1d64c9f817dd" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 41505 } } }
I20260924 03:20:57.338092 5431 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:20:57.367077 5438 catalog_manager.cc:1259] Loaded cluster ID: 210122c5105f480e93f1f7aa7cae2801
I20260924 03:20:57.367264 5438 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e: loading cluster ID for follower catalog manager: success
I20260924 03:20:57.370901 5438 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e: acquiring CA information for follower catalog manager: success
I20260924 03:20:57.373361 5438 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c52d1b7886854a578af6b7c6819abf0e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 03:20:57.480371 5228 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.464120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.14.227.190:42145)
0924 03:20:57.464312 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.464319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.478143 (+ 13824us) server_negotiation.cc:407] Connection header received
0924 03:20:57.478487 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.478496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.478648 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.478743 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.480113 (+ 1370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41812: BlockingRecv error: recv got EOF from 127.0.0.1:41812 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 03:20:57.488324 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.480120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.14.227.188:41505)
0924 03:20:57.480292 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.480297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.486373 (+ 6076us) server_negotiation.cc:407] Connection header received
0924 03:20:57.486925 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.486931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.487011 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.487103 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.488019 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32888: BlockingRecv error: recv got EOF from 127.0.0.1:32888 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 03:20:57.498883 5303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.470691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.14.227.189:45373)
0924 03:20:57.471010 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.471017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.486373 (+ 15356us) server_negotiation.cc:407] Connection header received
0924 03:20:57.486642 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.486651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.486733 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.486827 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.498641 (+ 11814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38432: BlockingRecv error: recv got EOF from 127.0.0.1:38432 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
I20260924 03:20:57.554837 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.536170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.14.227.189:45373)
0924 03:20:57.536340 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.536346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.546967 (+ 10621us) server_negotiation.cc:407] Connection header received
0924 03:20:57.547492 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.547498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.547716 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.547858 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:57.548571 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.549724 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.551739 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.552419 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.552534 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.553263 (+ 729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:57.554026 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:57.554137 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.554369 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.554482 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 03:20:57.608445 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.595199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.14.227.188:41505)
0924 03:20:57.595391 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.595400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.606606 (+ 11206us) server_negotiation.cc:407] Connection header received
0924 03:20:57.606849 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.606856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.606993 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.607130 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.608192 (+ 1062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32896: BlockingRecv error: recv got EOF from 127.0.0.1:32896 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 03:20:57.661247 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.642633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.14.227.189:45373)
0924 03:20:57.642891 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.642898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.653532 (+ 10634us) server_negotiation.cc:407] Connection header received
0924 03:20:57.654186 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.654194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.654294 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.654387 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:57.655147 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.656412 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.658671 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.659427 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.659559 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.660315 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:57.660486 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:57.660617 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.660803 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.660898 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 03:20:57.672369 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.669790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.14.227.188:41505)
0924 03:20:57.670018 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.670025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.670548 (+ 523us) server_negotiation.cc:407] Connection header received
0924 03:20:57.670824 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.670830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.670929 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.671022 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.672093 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32902: BlockingRecv error: recv got EOF from 127.0.0.1:32902 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:20:57.686902 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.678890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.14.227.189:45373)
0924 03:20:57.679106 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.679114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.679633 (+ 519us) server_negotiation.cc:407] Connection header received
0924 03:20:57.679807 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.679814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.679908 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.680024 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:57.680531 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.681762 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.684016 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.684844 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.684948 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.685804 (+ 856us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:57.686022 (+ 218us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:57.686094 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.686369 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.686460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 03:20:57.697157 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.694617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.14.227.188:41505)
0924 03:20:57.694809 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.694816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.695317 (+ 501us) server_negotiation.cc:407] Connection header received
0924 03:20:57.695618 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.695626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.695725 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.695807 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.696884 (+ 1077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32912: BlockingRecv error: recv got EOF from 127.0.0.1:32912 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 03:20:57.712306 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.705271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.14.227.189:45373)
0924 03:20:57.705487 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.705493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.706025 (+ 532us) server_negotiation.cc:407] Connection header received
0924 03:20:57.706227 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.706233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.706320 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.706424 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:57.706880 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.708032 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.710034 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.710755 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.710888 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.711426 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:57.711578 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:57.711676 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.711921 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.712003 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260924 03:20:57.723634 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.720788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.14.227.188:41505)
0924 03:20:57.721037 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.721044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.721647 (+ 603us) server_negotiation.cc:407] Connection header received
0924 03:20:57.722058 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.722068 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.722182 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.722267 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.723366 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32926: BlockingRecv error: recv got EOF from 127.0.0.1:32926 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 03:20:57.747531 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.739696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.14.227.189:45373)
0924 03:20:57.739902 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.739909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.740484 (+ 575us) server_negotiation.cc:407] Connection header received
0924 03:20:57.740708 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.740715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.740808 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.740923 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:57.741452 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.742757 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.745040 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.745792 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.745933 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.746593 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:57.746749 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:57.746853 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.747089 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.747179 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 03:20:57.757643 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.754952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.14.227.188:41505)
0924 03:20:57.755199 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.755207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.755759 (+ 552us) server_negotiation.cc:407] Connection header received
0924 03:20:57.756090 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.756098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.756215 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.756311 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.757392 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32938: BlockingRecv error: recv got EOF from 127.0.0.1:32938 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 03:20:57.786026 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.778417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.14.227.189:45373)
0924 03:20:57.778632 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.778639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.779233 (+ 594us) server_negotiation.cc:407] Connection header received
0924 03:20:57.779446 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.779452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.779535 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.779639 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:57.780247 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.781437 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.783643 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.784446 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.784571 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.785116 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:57.785252 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:57.785353 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.785552 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.785643 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 03:20:57.797479 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.794536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.14.227.188:41505)
0924 03:20:57.794734 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.794740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.795365 (+ 625us) server_negotiation.cc:407] Connection header received
0924 03:20:57.795747 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.795755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.795862 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.795984 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.797213 (+ 1229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32954: BlockingRecv error: recv got EOF from 127.0.0.1:32954 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 03:20:57.840307 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.834317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.14.227.189:45373)
0924 03:20:57.834554 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.834560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.834900 (+ 340us) server_negotiation.cc:407] Connection header received
0924 03:20:57.835175 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.835179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.835245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.835329 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:57.835870 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.836697 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.838315 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.838789 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.838871 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.839514 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:57.839621 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:57.839721 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.839878 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.839961 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260924 03:20:57.855248 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.853084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.14.227.188:41505)
0924 03:20:57.853352 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.853359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.853788 (+ 429us) server_negotiation.cc:407] Connection header received
0924 03:20:57.854024 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.854032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.854138 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.854272 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.854991 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32968: BlockingRecv error: recv got EOF from 127.0.0.1:32968 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 03:20:57.927999 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:57.922776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.14.227.189:45373)
0924 03:20:57.922942 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.922947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.923376 (+ 429us) server_negotiation.cc:407] Connection header received
0924 03:20:57.923599 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.923605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.923673 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.923774 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:57.924153 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.924891 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.926311 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:57.926745 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:57.926818 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:57.927281 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:57.927396 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:57.927516 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:57.927661 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 03:20:57.927739 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 03:20:57.935822 5375 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:20:57.933843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.14.227.188:41505)
0924 03:20:57.934110 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:57.934116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:57.934361 (+ 245us) server_negotiation.cc:407] Connection header received
0924 03:20:57.934524 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:57.934528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:57.934593 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:57.934687 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:20:57.935546 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32974: BlockingRecv error: recv got EOF from 127.0.0.1:32974 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:20:58.025729 5389 catalog_manager.cc:1259] Loaded cluster ID: 210122c5105f480e93f1f7aa7cae2801
I20260924 03:20:58.025835 5389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: loading cluster ID for follower catalog manager: success
I20260924 03:20:58.028342 5389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: acquiring CA information for follower catalog manager: success
I20260924 03:20:58.030082 5389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e369622cb3b4b55884a1d64c9f817dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:20:58.072314 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:58.067353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.14.227.189:45373)
0924 03:20:58.067600 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:58.067607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:58.067878 (+ 271us) server_negotiation.cc:407] Connection header received
0924 03:20:58.068039 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:58.068043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:58.068117 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:58.068199 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:58.068614 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.069330 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.070612 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.071030 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.071123 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:58.071517 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:58.071632 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:58.071703 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:58.071916 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:20:58.072034 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 03:20:58.083662 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:58.078104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.14.227.188:41505)
0924 03:20:58.078352 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:58.078358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:58.078597 (+ 239us) server_negotiation.cc:407] Connection header received
0924 03:20:58.078743 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:58.078747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:58.078900 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:58.078993 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:58.079301 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.079963 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.081430 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.081854 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.082011 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:58.082496 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:58.083060 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:58.083252 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:58.083306 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 03:20:58.083389 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 03:20:58.109117 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:58.090488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.14.227.189:45373)
0924 03:20:58.090643 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:58.090648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:58.102557 (+ 11909us) server_negotiation.cc:407] Connection header received
0924 03:20:58.102802 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:58.102810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:58.102917 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:58.103013 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:58.103724 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.104757 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.106855 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.107469 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.107585 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:58.108353 (+ 768us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:58.108465 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:58.108583 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:58.108738 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 03:20:58.108817 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 03:20:58.109488 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:58.090679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.14.227.188:41505)
0924 03:20:58.090917 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:58.090926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:58.103240 (+ 12314us) server_negotiation.cc:407] Connection header received
0924 03:20:58.103423 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:58.103431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:58.103536 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:58.103632 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:58.104162 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.105166 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.107122 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.107599 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.107719 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:58.108710 (+ 991us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:58.108822 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:58.108901 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:58.109141 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 03:20:58.109237 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 03:20:58.110656 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:20:58.090233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.14.227.190:42145)
0924 03:20:58.103763 (+ 13530us) server_negotiation.cc:206] Beginning negotiation
0924 03:20:58.103770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:20:58.103792 (+ 22us) server_negotiation.cc:407] Connection header received
0924 03:20:58.103909 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:20:58.103914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:20:58.104118 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:20:58.104347 (+ 229us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:20:58.104933 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.105747 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.107592 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:20:58.108084 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:20:58.108173 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:20:58.108683 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:20:58.109169 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:20:58.110287 (+ 1118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:20:58.110346 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 03:20:58.110414 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13393,"thread_start_us":13268,"threads_started":1}
I20260924 03:20:58.115375 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 5157
I20260924 03:20:58.116357 5219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:58.228148 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 5157
I20260924 03:20:58.248519 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 5231
I20260924 03:20:58.249553 5293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:58.355909 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 5231
I20260924 03:20:58.375353 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 5305
I20260924 03:20:58.376478 5367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:20:58.484963 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 5305
2026-09-24T03:20:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4381 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:20:58 dist-test-slave-vd0p krb5kdc[5553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:20:58 dist-test-slave-vd0p krb5kdc[5553](info): set up 2 sockets
Sep 24 03:20:58 dist-test-slave-vd0p krb5kdc[5553](info): commencing operation
krb5kdc: starting...
W20260924 03:20:59.602553 15246 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/test-user.keytab.
Sep 24 03:20:59 dist-test-slave-vd0p krb5kdc[5553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.227.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
Entry for principal kudu/127.14.227.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab.
I20260924 03:20:59.854969 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:40775
--webserver_interface=127.14.227.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:40775,127.14.227.189:33849,127.14.227.188:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:21:00.174212 5566 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:21:00.174558 5566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:00.174697 5566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:00.174818 5566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:00.181993 5566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:00.182076 5566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:00.182180 5566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:00.182282 5566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:00.182370 5566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:00.192306 5566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:40775,127.14.227.189:33849,127.14.227.188:33845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.190.keytab
--principal=kudu/127.14.227.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.190:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.227.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:00.193594 5566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:00.194900 5566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:00.206583 5571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:00.206629 5574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:00.206545 5572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:00.207428 5566 server_base.cc:1034] running on GCE node
I20260924 03:21:00.208307 5566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:00.209705 5566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:00.210951 5566 hybrid_clock.cc:648] HybridClock initialized: now 1790220060210866 us; error 51 us; skew 500 ppm
I20260924 03:21:00.211321 5566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:00 dist-test-slave-vd0p krb5kdc[5553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:00.217777 5566 init.cc:377] Logged in from keytab as kudu/127.14.227.190@KRBTEST.COM (short username kudu)
I20260924 03:21:00.219285 5566 webserver.cc:460] Webserver started at http://127.14.227.190:44067/ using document root <none> and password file <none>
I20260924 03:21:00.219998 5566 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:00.220112 5566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:00.220445 5566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:00.223487 5566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/data/instance:
uuid: "0d63f2ff77fa4c6cab1709eeffeadc71"
format_stamp: "Formatted at 2026-09-24 03:21:00 on dist-test-slave-vd0p"
server_key: "3f93b85829dd1321431f57db4ff706e7"
server_key_iv: "dae46cd96b64cffba67ca06280af8969"
server_key_version: "encryptionkey@0"
I20260924 03:21:00.224313 5566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/wal/instance:
uuid: "0d63f2ff77fa4c6cab1709eeffeadc71"
format_stamp: "Formatted at 2026-09-24 03:21:00 on dist-test-slave-vd0p"
server_key: "3f93b85829dd1321431f57db4ff706e7"
server_key_iv: "dae46cd96b64cffba67ca06280af8969"
server_key_version: "encryptionkey@0"
I20260924 03:21:00.229578 5566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 03:21:00.232951 5581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:00.234301 5566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 03:21:00.234481 5566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/wal
uuid: "0d63f2ff77fa4c6cab1709eeffeadc71"
format_stamp: "Formatted at 2026-09-24 03:21:00 on dist-test-slave-vd0p"
server_key: "3f93b85829dd1321431f57db4ff706e7"
server_key_iv: "dae46cd96b64cffba67ca06280af8969"
server_key_version: "encryptionkey@0"
I20260924 03:21:00.234681 5566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:00.249034 5566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:00.253228 5566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:00.253497 5566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:00.271102 5566 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.190:40775
I20260924 03:21:00.271121 5632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.190:40775 every 8 connection(s)
I20260924 03:21:00.272418 5566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/data/info.pb
I20260924 03:21:00.275974 5633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:00.279448 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 5566
I20260924 03:21:00.279631 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-0/wal/instance
I20260924 03:21:00.280150 15246 external_mini_cluster.cc:1428] Setting key 15b9927203f7390b69357df165dd2ccd
I20260924 03:21:00.284075 5633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:21:00 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.227.189@KRBTEST.COM; defaulting to no policy
I20260924 03:21:00.308643 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.287746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.14.227.190:40775)
0924 03:21:00.289660 (+ 1914us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:00.289673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:00.289750 (+ 77us) server_negotiation.cc:407] Connection header received
0924 03:21:00.291330 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:00.291341 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:00.291643 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:00.291858 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:00.292941 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.294909 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.296238 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.297180 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.297361 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:00.300609 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:00.300622 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:00.300641 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:00.300684 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:00.304737 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:00.305393 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:00.305403 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:00.305408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:00.305593 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:00.305981 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:00.305987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:00.305994 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:00.306688 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:00.306932 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:00.307369 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 03:21:00.307779 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":1667,"spinlock_wait_cycles":14208,"thread_start_us":1442,"threads_started":1}
I20260924 03:21:00.308681 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.287746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:40775 (local address 127.0.0.1:48944)
0924 03:21:00.288649 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.288716 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:00.290961 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:00.291881 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:00.291886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:00.292413 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:00.292730 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:00.292745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.295052 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.295055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:00.296098 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:00.296108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.297428 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.297432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:00.297551 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:00.298249 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:00.298271 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:00.300374 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:00.304883 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:00.304891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:00.304899 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:00.305254 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:00.305709 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:00.305712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:00.305714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:00.305857 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:00.306825 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:00.306831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:00.307004 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 03:21:00.307193 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 03:21:00.307611 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":370,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.189@KRBTEST.COM" created.
I20260924 03:21:00.311496 5633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:21:00.312413 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.312057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43912)
0924 03:21:00.312210 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.312314 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 03:21:00.312618 5582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
I20260924 03:21:00.314142 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.313746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43918)
0924 03:21:00.313896 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.313997 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 03:21:00.314349 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:33849: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab.
Entry for principal kudu/127.14.227.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab.
I20260924 03:21:00.338999 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.189:33849
--webserver_interface=127.14.227.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab
--principal=kudu/127.14.227.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:40775,127.14.227.189:33849,127.14.227.188:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:21:00.363649 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } attempt: 1
I20260924 03:21:00.364750 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.364320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43934)
0924 03:21:00.364508 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.364613 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 03:21:00.365870 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.365454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43942)
0924 03:21:00.365620 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.365706 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 03:21:00.366067 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:33849: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
I20260924 03:21:00.434355 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } attempt: 2
I20260924 03:21:00.435806 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.435156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43948)
0924 03:21:00.435472 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.435610 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 03:21:00.436893 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.436485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43960)
0924 03:21:00.436636 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.436742 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:21:00.437129 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:33849: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
I20260924 03:21:00.528399 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } attempt: 3
I20260924 03:21:00.529838 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.529242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43964)
0924 03:21:00.529511 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.529632 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 03:21:00.531070 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.530737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43978)
0924 03:21:00.530878 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.530953 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 03:21:00.531250 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:33849: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
W20260924 03:21:00.665144 5640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:21:00.665527 5640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:00.665660 5640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:00.665804 5640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:00.672837 5640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:00.672963 5640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:00.673074 5640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:00.673166 5640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:00.673262 5640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:00.674489 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } attempt: 4
I20260924 03:21:00.675668 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.675138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43986)
0924 03:21:00.675439 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.675536 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260924 03:21:00.676733 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.676372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:43994)
0924 03:21:00.676527 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.676617 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 03:21:00.676936 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.189:33849: Network error: Client connection negotiation failed: client connection to 127.14.227.189:33849: connect: Connection refused (error 111)
I20260924 03:21:00.683297 5640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:40775,127.14.227.189:33849,127.14.227.188:33845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.189.keytab
--principal=kudu/127.14.227.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.189:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.227.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:00.684535 5640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:00.685832 5640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:21:00.697525 5645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:00.697484 5648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:00.697558 5646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:00.698422 5640 server_base.cc:1034] running on GCE node
I20260924 03:21:00.699191 5640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:00.700507 5640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:00.701741 5640 hybrid_clock.cc:648] HybridClock initialized: now 1790220060701671 us; error 45 us; skew 500 ppm
I20260924 03:21:00.702107 5640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:00 dist-test-slave-vd0p krb5kdc[5553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:00.705018 5640 init.cc:377] Logged in from keytab as kudu/127.14.227.189@KRBTEST.COM (short username kudu)
I20260924 03:21:00.706449 5640 webserver.cc:460] Webserver started at http://127.14.227.189:42723/ using document root <none> and password file <none>
I20260924 03:21:00.707127 5640 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:00.707255 5640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:00.707559 5640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:00.710425 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/data/instance:
uuid: "14c178677eae4d42abc82adc00f7f38b"
format_stamp: "Formatted at 2026-09-24 03:21:00 on dist-test-slave-vd0p"
server_key: "26f527709f2c8bbf33204086387542a6"
server_key_iv: "6571e6454b14b00d063de2cc97dd54f1"
server_key_version: "encryptionkey@0"
I20260924 03:21:00.711190 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/wal/instance:
uuid: "14c178677eae4d42abc82adc00f7f38b"
format_stamp: "Formatted at 2026-09-24 03:21:00 on dist-test-slave-vd0p"
server_key: "26f527709f2c8bbf33204086387542a6"
server_key_iv: "6571e6454b14b00d063de2cc97dd54f1"
server_key_version: "encryptionkey@0"
I20260924 03:21:00.716109 5640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 03:21:00.719513 5655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:00.720659 5640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 03:21:00.720836 5640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/wal
uuid: "14c178677eae4d42abc82adc00f7f38b"
format_stamp: "Formatted at 2026-09-24 03:21:00 on dist-test-slave-vd0p"
server_key: "26f527709f2c8bbf33204086387542a6"
server_key_iv: "6571e6454b14b00d063de2cc97dd54f1"
server_key_version: "encryptionkey@0"
I20260924 03:21:00.721040 5640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:00.737464 5640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:00.741446 5640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:00.741710 5640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:00.759680 5640 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.189:33849
I20260924 03:21:00.759690 5706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.189:33849 every 8 connection(s)
I20260924 03:21:00.761053 5640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/data/info.pb
I20260924 03:21:00.762692 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 5640
I20260924 03:21:00.762880 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-1/wal/instance
I20260924 03:21:00.763329 15246 external_mini_cluster.cc:1428] Setting key 0cdf0d5ab506a195190a6aac125f688c
I20260924 03:21:00.764850 5707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:00.771683 5707 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:21:00 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.227.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.227.188@KRBTEST.COM" created.
I20260924 03:21:00.792718 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.775041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.14.227.190:40775)
0924 03:21:00.775263 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:00.775269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:00.776486 (+ 1217us) server_negotiation.cc:407] Connection header received
0924 03:21:00.778332 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:00.778339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:00.778430 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:00.778548 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:00.779932 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.781058 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.782582 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.783013 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.783163 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:00.786825 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:00.786830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:00.786834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:00.786866 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:00.789569 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:00.790395 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:00.790399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:00.790402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:00.790540 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:00.790976 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:00.790981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:00.790986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:00.791271 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:00.791434 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:00.792142 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 03:21:00.792391 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 03:21:00.793666 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.775504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:40775 (local address 127.0.0.1:48958)
0924 03:21:00.776282 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.776354 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:00.778082 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:00.778722 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:00.778729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:00.779211 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:00.779748 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:00.779763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.781200 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.781203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:00.782440 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:00.782452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.783237 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.783244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:00.783472 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:00.784178 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:00.784198 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:00.786603 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:00.789714 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:00.789722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:00.789739 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:00.790257 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:00.790688 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:00.790691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:00.790694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:00.790854 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:00.791409 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:00.791417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:00.791762 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 03:21:00.792032 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 03:21:00.792563 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":339,"threads_started":1}
I20260924 03:21:00.795389 5707 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:21:00 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:21:00.810195 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.796318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.14.227.189:33849)
0924 03:21:00.796929 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:00.796934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:00.796958 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:21:00.797000 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:00.797006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:00.797169 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:00.797348 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:00.798061 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.799082 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.800182 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.800700 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.800833 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:00.804460 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:00.804470 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:00.804483 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:00.804509 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:00.807148 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:00.807999 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:00.808004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:00.808007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:00.808108 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:00.808534 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:00.808539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:00.808542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:00.808868 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:00.809017 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:00.809487 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 03:21:00.809829 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":478,"thread_start_us":340,"threads_started":1}
I20260924 03:21:00.810320 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.796022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:44002)
0924 03:21:00.796237 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.796251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:00.796336 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:00.797309 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:00.797313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:00.797748 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:00.797876 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:00.797884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.799210 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.799217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:00.800052 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:00.800062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.800847 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.800852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:00.801040 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:00.801655 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:00.801681 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:00.804247 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:00.807292 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:00.807300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:00.807307 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:00.807881 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:00.808214 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:00.808222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:00.808229 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:00.808409 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:00.809008 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:00.809016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:00.809189 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 03:21:00.809367 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 03:21:00.809708 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260924 03:21:00.812563 5707 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:21:00.813644 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.813200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44550)
0924 03:21:00.813363 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.813499 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 03:21:00.813913 5659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
I20260924 03:21:00.815467 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.815088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44552)
0924 03:21:00.815257 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.815328 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:21:00.815683 5707 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.227.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab.
Entry for principal kudu/127.14.227.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab.
I20260924 03:21:00.820997 15246 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1C3avF/build/asan/bin/kudu
/tmp/dist-test-task1C3avF/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.188:33845
--webserver_interface=127.14.227.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab
--principal=kudu/127.14.227.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.227.148:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.227.190:40775,127.14.227.189:33849,127.14.227.188:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:21:00.865012 5707 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 1
I20260924 03:21:00.866243 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.865667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44566)
0924 03:21:00.865906 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.866057 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 03:21:00.867260 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.866858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44570)
0924 03:21:00.867032 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.867127 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 03:21:00.867486 5707 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
I20260924 03:21:00.935738 5707 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 2
I20260924 03:21:00.936929 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.936400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44580)
0924 03:21:00.936618 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.936737 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 03:21:00.938014 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.937594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44594)
0924 03:21:00.937750 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.937832 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:21:00.938270 5707 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
I20260924 03:21:00.976159 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } attempt: 5
Sep 24 03:21:00 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:21:00.991103 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.977036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:44004)
0924 03:21:00.977263 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.977285 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:00.977466 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:00.977838 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:00.977842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:00.978284 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:00.978472 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:00.978481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.979470 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.979477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:00.980426 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:00.980439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.981194 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.981200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:00.981398 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:00.982112 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:00.982130 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:00.984359 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:00.988433 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:00.988437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:00.988439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:00.988792 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:00.989247 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:00.989250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:00.989252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:00.989389 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:00.990183 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:00.990186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:00.990315 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 03:21:00.990442 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 03:21:00.990636 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 03:21:00.991421 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.977073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.14.227.189:33849)
0924 03:21:00.977263 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:00.977270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:00.977390 (+ 120us) server_negotiation.cc:407] Connection header received
0924 03:21:00.977625 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:00.977629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:00.977693 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:00.977799 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:00.978624 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.979338 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.980580 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:00.981029 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:00.981160 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:00.984579 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:00.984587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:00.984592 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:00.984638 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:00.988211 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:00.988922 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:00.988931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:00.988938 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:00.989121 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:00.989510 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:00.989519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:00.989525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:00.990053 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:00.990246 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:00.990640 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 03:21:00.990946 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 03:21:00.992540 5633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:21:00.993794 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.993235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44598)
0924 03:21:00.993453 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.993596 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 03:21:00.994849 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:00.994433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44614)
0924 03:21:00.994596 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:00.994721 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 03:21:00.995090 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
I20260924 03:21:01.029515 5707 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 3
I20260924 03:21:01.030779 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.030174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44618)
0924 03:21:01.030408 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.030542 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 03:21:01.031952 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.031453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44626)
0924 03:21:01.031712 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.031811 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260924 03:21:01.032199 5707 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
I20260924 03:21:01.046303 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 1
I20260924 03:21:01.047468 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.046861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44636)
0924 03:21:01.047116 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.047239 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 03:21:01.048609 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.048169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44642)
0924 03:21:01.048349 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.048449 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 03:21:01.048833 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
I20260924 03:21:01.117101 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 2
I20260924 03:21:01.118530 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.117849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44656)
0924 03:21:01.118150 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.118282 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 03:21:01.119673 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.119234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44672)
0924 03:21:01.119406 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.119522 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 03:21:01.119947 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
W20260924 03:21:01.155534 5714 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:21:01.155860 5714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:21:01.155951 5714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:21:01.156039 5714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:21:01.163270 5714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:21:01.163371 5714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:21:01.163436 5714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:21:01.163494 5714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:21:01.163571 5714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:21:01.173280 5714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.227.148:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.227.190:40775,127.14.227.189:33849,127.14.227.188:33845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/krb5kdc/kudu_127.14.227.188.keytab
--principal=kudu/127.14.227.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.227.188:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.227.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 02:13:03 UTC on ccycloud.quasar-nitgar.root.comops.site
build id 4669252
ASAN enabled
I20260924 03:21:01.174647 5714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:21:01.175406 5707 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 4
I20260924 03:21:01.176092 5714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:21:01.176599 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.175989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44680)
0924 03:21:01.176266 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.176388 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 03:21:01.177644 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.177261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44694)
0924 03:21:01.177417 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.177504 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 03:21:01.177878 5707 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
W20260924 03:21:01.188241 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:01.188362 5720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:21:01.188284 5719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:21:01.189368 5714 server_base.cc:1034] running on GCE node
I20260924 03:21:01.190184 5714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:21:01.191620 5714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:21:01.192860 5714 hybrid_clock.cc:648] HybridClock initialized: now 1790220061192778 us; error 55 us; skew 500 ppm
I20260924 03:21:01.193212 5714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220061, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:21:01.196170 5714 init.cc:377] Logged in from keytab as kudu/127.14.227.188@KRBTEST.COM (short username kudu)
I20260924 03:21:01.197502 5714 webserver.cc:460] Webserver started at http://127.14.227.188:46001/ using document root <none> and password file <none>
I20260924 03:21:01.198225 5714 fs_manager.cc:359] Metadata directory not provided
I20260924 03:21:01.198352 5714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:21:01.198650 5714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:21:01.201424 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/data/instance:
uuid: "8628b971804d4f9cbdb8db9b74cfcec2"
format_stamp: "Formatted at 2026-09-24 03:21:01 on dist-test-slave-vd0p"
server_key: "42ca7f7850a489d38fa8ec4446ccec08"
server_key_iv: "dd62fcfd37f3c5e456125546757a40cd"
server_key_version: "encryptionkey@0"
I20260924 03:21:01.202241 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/wal/instance:
uuid: "8628b971804d4f9cbdb8db9b74cfcec2"
format_stamp: "Formatted at 2026-09-24 03:21:01 on dist-test-slave-vd0p"
server_key: "42ca7f7850a489d38fa8ec4446ccec08"
server_key_iv: "dd62fcfd37f3c5e456125546757a40cd"
server_key_version: "encryptionkey@0"
I20260924 03:21:01.207125 5714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 03:21:01.210392 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:21:01.211661 5714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 03:21:01.211846 5714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/data,/tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/wal
uuid: "8628b971804d4f9cbdb8db9b74cfcec2"
format_stamp: "Formatted at 2026-09-24 03:21:01 on dist-test-slave-vd0p"
server_key: "42ca7f7850a489d38fa8ec4446ccec08"
server_key_iv: "dd62fcfd37f3c5e456125546757a40cd"
server_key_version: "encryptionkey@0"
I20260924 03:21:01.212054 5714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:21:01.226154 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 3
I20260924 03:21:01.227262 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.226806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44700)
0924 03:21:01.227018 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.227104 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 03:21:01.228215 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.227865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44710)
0924 03:21:01.228050 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.228113 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 03:21:01.228425 5633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.227.188:33845: Network error: Client connection negotiation failed: client connection to 127.14.227.188:33845: connect: Connection refused (error 111)
I20260924 03:21:01.254382 5714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:21:01.258368 5714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:21:01.258616 5714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:21:01.277209 5714 rpc_server.cc:307] RPC server started. Bound to: 127.14.227.188:33845
I20260924 03:21:01.277244 5780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.227.188:33845 every 8 connection(s)
I20260924 03:21:01.278548 5714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/data/info.pb
I20260924 03:21:01.282226 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:21:01.284854 15246 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1C3avF/build/asan/bin/kudu as pid 5714
I20260924 03:21:01.285036 15246 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1C3avF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790219707325830-15246-0/minicluster-data/master-2/wal/instance
I20260924 03:21:01.285475 15246 external_mini_cluster.cc:1428] Setting key 68e055527a8ea3f9a582c66e6ce6c622
I20260924 03:21:01.289415 5781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220061, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:21:01.307241 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.291978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.14.227.190:40775)
0924 03:21:01.292202 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.292209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.292886 (+ 677us) server_negotiation.cc:407] Connection header received
0924 03:21:01.293068 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.293073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.293133 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.293216 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.294639 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.295692 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.297590 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.298237 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.298365 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.301906 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.301911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.301914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.301988 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.304467 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.305116 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.305121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.305125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.305255 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.305636 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.305642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.305646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.305895 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.306100 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.306656 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.306872 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:21:01.308245 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.291954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.190:40775 (local address 127.0.0.1:48964)
0924 03:21:01.292747 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.292787 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:01.292934 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:01.293306 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:01.293314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:01.294133 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:01.294494 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.294505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.295847 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.295850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.297440 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.297453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.298449 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.298454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.298571 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.299151 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:01.299182 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:01.301717 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:01.304596 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.304608 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.304620 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.304996 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.305364 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.305368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.305372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.305529 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.306073 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:01.306081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:01.306287 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 03:21:01.306561 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 03:21:01.307060 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":32,"thread_start_us":485,"threads_started":1}
I20260924 03:21:01.308264 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.287282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.14.227.188:33845)
0924 03:21:01.288023 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.288034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.288071 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:21:01.289733 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.289753 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.290224 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.290580 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.292233 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.294325 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.296060 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.297080 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.297262 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.301398 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.301406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.301417 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.301446 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.304428 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.305077 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.305093 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.305095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.305232 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.305600 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.305606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.305609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.306184 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.306493 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.306848 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.307383 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":556,"thread_start_us":329,"threads_started":1}
I20260924 03:21:01.309749 5781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220061, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.190@KRBTEST.COM
I20260924 03:21:01.324082 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.310407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.189:33849 (local address 127.0.0.1:44020)
0924 03:21:01.310534 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.310553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:01.310644 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:01.310986 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:01.310995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:01.311469 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:01.311662 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.311673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.312659 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.312665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.313619 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.313630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.314424 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.314430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.314644 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.315891 (+ 1247us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:01.315913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:01.318263 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:01.321670 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.321676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.321680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.322103 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.322472 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.322477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.322481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.322619 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.323135 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:01.323140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:01.323300 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 03:21:01.323457 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 03:21:01.323729 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:21:01.324363 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.310535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.14.227.189:33849)
0924 03:21:01.310705 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.310709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.310724 (+ 15us) server_negotiation.cc:407] Connection header received
0924 03:21:01.310771 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.310775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.310842 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.310943 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.311815 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.312523 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.313790 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.314277 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.314385 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.318451 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.318456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.318460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.318491 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.321538 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.322220 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.322232 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.322234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.322359 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.322724 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.322730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.322734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.323020 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.323175 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.323554 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.323799 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:21:01.325320 5781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:21:01.325716 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.311276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.14.227.190:40775)
0924 03:21:01.311419 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.311423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.311462 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:21:01.311621 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.311625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.311681 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.311784 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.312887 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.313596 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.315159 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.315552 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.315678 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.320369 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.320377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.320383 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.320425 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.323039 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.323673 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.323679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.323682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.323807 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.324178 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.324183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.324187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.324563 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.324830 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.325167 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.325423 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220061, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.188@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.227.189@KRBTEST.COM
I20260924 03:21:01.339313 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.326508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.14.227.188:33845)
0924 03:21:01.326647 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.326651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.326663 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:21:01.326725 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.326731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.326794 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.326891 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.327654 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.328716 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.330021 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.330421 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.330574 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.334039 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.334045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.334050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.334089 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.336839 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.337438 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.337443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.337448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.337564 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.337920 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.337926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.337929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.338230 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.338378 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.338709 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.338942 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":57}
I20260924 03:21:01.339426 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.326156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:44720)
0924 03:21:01.326316 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.326336 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:01.326450 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:01.326897 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:01.326901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:01.327344 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:01.327529 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.327539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.328839 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.328843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.329838 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.329852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.330587 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.330591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.330785 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.331343 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:01.331367 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:01.333818 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:01.336973 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.336981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.336986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.337306 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.337676 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.337681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.337685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.337816 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.338343 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:01.338347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:01.338493 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 03:21:01.338638 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 03:21:01.339077 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":161}
I20260924 03:21:01.342753 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.327366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.14.227.189:33849)
0924 03:21:01.327514 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.327518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.327867 (+ 349us) server_negotiation.cc:407] Connection header received
0924 03:21:01.328004 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.328007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.328057 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.328129 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.329501 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.330382 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.331604 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.332234 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.332359 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.336885 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.336890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.336893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.336920 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.340199 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.340893 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.340898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.340901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.341066 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.341411 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.341415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.341418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.341732 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.342037 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.342214 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.342441 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:21:01.345696 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: Bootstrap starting.
I20260924 03:21:01.347388 15246 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:21:01.349215 5781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:01.350620 5781 log.cc:824] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:01.349709 5608 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48970
I20260924 03:21:01.353570 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: No bootstrap required, opened a new log
I20260924 03:21:01.364065 5781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } }
I20260924 03:21:01.364471 5781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8628b971804d4f9cbdb8db9b74cfcec2, State: Initialized, Role: FOLLOWER
I20260924 03:21:01.364975 5781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } }
I20260924 03:21:01.365837 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.366214 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:21:01.367096 5781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 03:21:01.376470 5798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:21:01.376595 5798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:21:01.373850 5714 master_runner.cc:186] Error getting master registration for 127.14.227.190:40775: OK, instance_id { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" instance_seqno: 1790220060266164 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:21:01.385087 5714 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:21:01.405611 5633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 4
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.190@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
I20260924 03:21:01.422535 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.406603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.14.227.188:33845)
0924 03:21:01.406823 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.406830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.406851 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:21:01.406933 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.406943 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.407024 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.407140 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.408079 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.409242 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.411067 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.411769 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.411915 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.415845 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.415853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.415858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.415895 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.419513 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.420315 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.420320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.420324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.420463 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.420888 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.420894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.420898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.421323 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.421486 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.421935 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.422163 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:21:01.422545 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.406382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:57670)
0924 03:21:01.406602 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.406623 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:01.406750 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:01.407182 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:01.407187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:01.407694 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:01.407911 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.407923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.409387 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.409392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.410918 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.410938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.411931 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.411936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.412134 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.412799 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:01.412820 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:01.415620 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:01.419651 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.419657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.419661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.420184 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.420575 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.420581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.420587 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.420777 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.421464 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:01.421469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:01.421665 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 03:21:01.421827 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 03:21:01.422152 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 03:21:01.428386 5633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71: Bootstrap starting.
I20260924 03:21:01.431552 5633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:01.432585 5633 log.cc:824] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:01.435504 5633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71: No bootstrap required, opened a new log
I20260924 03:21:01.441072 5633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } }
I20260924 03:21:01.441416 5633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d63f2ff77fa4c6cab1709eeffeadc71, State: Initialized, Role: FOLLOWER
I20260924 03:21:01.441916 5633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } }
I20260924 03:21:01.442885 5801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.443313 5801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:21:01.444084 5633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:01.444207 5608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:21:01.444420 5608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:01.447906 5608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } }
I20260924 03:21:01.450251 5756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8628b971804d4f9cbdb8db9b74cfcec2"
I20260924 03:21:01.450740 5756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:01.451112 5608 leader_election.cc:288] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [CANDIDATE]: Term 1 election: Requested vote from peers 14c178677eae4d42abc82adc00f7f38b (127.14.227.189:33849), 8628b971804d4f9cbdb8db9b74cfcec2 (127.14.227.188:33845)
I20260924 03:21:01.454679 5756 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d63f2ff77fa4c6cab1709eeffeadc71 in term 1.
I20260924 03:21:01.455287 5585 leader_election.cc:302] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d63f2ff77fa4c6cab1709eeffeadc71, 8628b971804d4f9cbdb8db9b74cfcec2; no voters:
I20260924 03:21:01.455751 5801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:21:01.466307 5801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [term 1 LEADER]: Becoming Leader. State: Replica: 0d63f2ff77fa4c6cab1709eeffeadc71, State: Running, Role: LEADER
I20260924 03:21:01.467072 5801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } }
I20260924 03:21:01.472690 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d63f2ff77fa4c6cab1709eeffeadc71. Latest consensus state: current_term: 1 leader_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.473124 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:01.477120 5707 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } attempt: 5
I20260924 03:21:01.477603 5812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:21:01.490326 5812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 03:21:01 dist-test-slave-vd0p krb5kdc[5553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790220060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.227.189@KRBTEST.COM for kudu/127.14.227.188@KRBTEST.COM
I20260924 03:21:01.502178 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.477811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.227.188:33845 (local address 127.0.0.1:57684)
0924 03:21:01.478124 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 03:21:01.478146 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 03:21:01.478263 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:21:01.478969 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:21:01.478977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:21:01.479648 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:21:01.479918 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.479932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.481835 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.481841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.483749 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:21:01.483769 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.487756 (+ 3987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.487765 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:21:01.488096 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.488862 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:21:01.488892 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:21:01.493058 (+ 4166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:21:01.498447 (+ 5389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.498455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.498458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.499118 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.499676 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:21:01.499684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:21:01.499690 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:21:01.499934 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:21:01.500809 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:21:01.500817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:21:01.501025 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 03:21:01.501210 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 03:21:01.501606 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260924 03:21:01.503475 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.478081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.14.227.188:33845)
0924 03:21:01.478297 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.478305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.478333 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:21:01.478410 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.478416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.478542 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.478681 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.480178 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.481625 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.483969 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.484897 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.487413 (+ 2516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.493373 (+ 5960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.493383 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.493390 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.493452 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.498215 (+ 4763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.499297 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.499306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.499312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.499521 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.500087 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.500094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.500100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.500525 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.500699 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.502608 (+ 1909us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.502932 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 03:21:01.506204 5587 tablet.cc:2369] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:01.509727 5707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b: Bootstrap starting.
W20260924 03:21:01.510618 5735 tablet.cc:2369] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:21:01.514873 5707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b: Neither blocks nor log segments found. Creating new log.
I20260924 03:21:01.516618 5707 log.cc:824] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:21:01.533941 5707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b: No bootstrap required, opened a new log
I20260924 03:21:01.540369 5756 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [term 1 FOLLOWER]: Refusing update from remote peer 0d63f2ff77fa4c6cab1709eeffeadc71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:01.543491 5809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:01.558000 5707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } }
I20260924 03:21:01.558688 5707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c178677eae4d42abc82adc00f7f38b, State: Initialized, Role: FOLLOWER
I20260924 03:21:01.559494 5707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } }
I20260924 03:21:01.563393 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d63f2ff77fa4c6cab1709eeffeadc71. Latest consensus state: current_term: 1 leader_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.563697 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:21:01.570494 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.518457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.14.227.188:33845)
0924 03:21:01.518721 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.518731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.521074 (+ 2343us) server_negotiation.cc:407] Connection header received
0924 03:21:01.521282 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.521289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.521400 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.521572 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.524479 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.525883 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.530328 (+ 4445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.531222 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.531346 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.546063 (+ 14717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.546074 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.546081 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.546138 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.551342 (+ 5204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.559055 (+ 7713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.559061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.559065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.559230 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.567517 (+ 8287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.567524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.567530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.568047 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.568355 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.569584 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.569977 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 03:21:01.570668 5826 mvcc.cc:205] Tried to move back new op lower bound from 7332741372030963712 to 7332741371775582208. Current Snapshot: MvccSnapshot[applied={T|T < 7332741372030963712}]
I20260924 03:21:01.581890 5818 mvcc.cc:205] Tried to move back new op lower bound from 7332741372030963712 to 7332741371775582208. Current Snapshot: MvccSnapshot[applied={T|T < 7332741372030963712}]
I20260924 03:21:01.553107 5566 master_runner.cc:186] Error getting master registration for 127.14.227.189:33849: OK, instance_id { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" instance_seqno: 1790220060754501 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:21:01.582587 5828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.583098 5828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:21:01.585131 5682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "14c178677eae4d42abc82adc00f7f38b"
I20260924 03:21:01.585573 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.520163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.14.227.189:33849)
0924 03:21:01.520334 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.520340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.522206 (+ 1866us) server_negotiation.cc:407] Connection header received
0924 03:21:01.524569 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.524575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.524655 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.524724 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.535865 (+ 11141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.537026 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.549460 (+ 12434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.550156 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.550271 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.575950 (+ 25679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.575960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.575968 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.576020 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.580761 (+ 4741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.582853 (+ 2092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.582860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.582863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.583113 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.583597 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.583604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.583608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.584044 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.584311 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.584871 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.585141 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 03:21:01.585507 5681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:21:01.585853 5707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:21:01.590423 5681 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [term 1 FOLLOWER]: Refusing update from remote peer 0d63f2ff77fa4c6cab1709eeffeadc71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:21:01.595010 5682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d63f2ff77fa4c6cab1709eeffeadc71 in term 1.
I20260924 03:21:01.595480 5566 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:21:01.596551 5801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:21:01.607684 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.508924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.14.227.190:40775)
0924 03:21:01.509106 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.509112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.519519 (+ 10407us) server_negotiation.cc:407] Connection header received
0924 03:21:01.522496 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.522502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.522584 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.522656 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.532834 (+ 10178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.533970 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.553759 (+ 19789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.554555 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.554683 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.592218 (+ 37535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:21:01.592226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:21:01.592232 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:21:01.592281 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:21:01.596165 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.597910 (+ 1745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.597915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.597919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.598180 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:21:01.605773 (+ 7593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:21:01.605780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:21:01.605784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:21:01.606211 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:21:01.606456 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.607060 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.607319 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 03:21:01.612571 5812 catalog_manager.cc:1347] Generated new cluster ID: c2cd013b50ab4077851445016d842a19
I20260924 03:21:01.612659 5812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:21:01.625344 5801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.625656 5801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:01.639806 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.640071 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:21:01.689167 5812 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:21:01.691043 5812 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:21:01.692979 5828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d63f2ff77fa4c6cab1709eeffeadc71. Latest consensus state: current_term: 1 leader_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.693442 5828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:21:01.694437 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.694898 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:21:01.696579 5801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d63f2ff77fa4c6cab1709eeffeadc71" member_type: VOTER last_known_addr { host: "127.14.227.190" port: 40775 } } peers { permanent_uuid: "14c178677eae4d42abc82adc00f7f38b" member_type: VOTER last_known_addr { host: "127.14.227.189" port: 33849 } } peers { permanent_uuid: "8628b971804d4f9cbdb8db9b74cfcec2" member_type: VOTER last_known_addr { host: "127.14.227.188" port: 33845 } } }
I20260924 03:21:01.696941 5801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71 [sys.catalog]: This master's current role is: LEADER
I20260924 03:21:01.713742 5812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d63f2ff77fa4c6cab1709eeffeadc71: Generated new TSK 0
I20260924 03:21:01.714916 5812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:21:01.737183 5846 catalog_manager.cc:1259] Loaded cluster ID: c2cd013b50ab4077851445016d842a19
I20260924 03:21:01.737303 5846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b: loading cluster ID for follower catalog manager: success
I20260924 03:21:01.741293 5846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b: acquiring CA information for follower catalog manager: success
I20260924 03:21:01.743891 5846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14c178677eae4d42abc82adc00f7f38b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 03:21:01.853989 5636 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:01.834512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.14.227.190:40775)
0924 03:21:01.834740 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.834747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.848436 (+ 13689us) server_negotiation.cc:407] Connection header received
0924 03:21:01.851120 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.851127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.851222 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.851309 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.853683 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40204: BlockingRecv error: recv got EOF from 127.0.0.1:40204 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 03:21:01.857597 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:01.843340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.14.227.188:33845)
0924 03:21:01.843546 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.843552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.855075 (+ 11523us) server_negotiation.cc:407] Connection header received
0924 03:21:01.856209 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.856217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.856311 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.856388 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.857353 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57692: BlockingRecv error: recv got EOF from 127.0.0.1:57692 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 03:21:01.860117 5711 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:01.848986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.14.227.189:33849)
0924 03:21:01.849204 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.849210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.858466 (+ 9256us) server_negotiation.cc:407] Connection header received
0924 03:21:01.858733 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.858739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.858842 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.858917 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.859871 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36236: BlockingRecv error: recv got EOF from 127.0.0.1:36236 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 03:21:01.914155 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.894136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.14.227.189:33849)
0924 03:21:01.894351 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.894357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.906538 (+ 12181us) server_negotiation.cc:407] Connection header received
0924 03:21:01.906754 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.906760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.906988 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.907160 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:01.907850 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.909022 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.911075 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.911762 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.911856 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.912603 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:01.913287 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:01.913434 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.913674 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.913777 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 03:21:01.952075 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:01.946192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.14.227.188:33845)
0924 03:21:01.946379 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.946386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.950509 (+ 4123us) server_negotiation.cc:407] Connection header received
0924 03:21:01.950710 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.950716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.950853 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.950970 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.951828 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57700: BlockingRecv error: recv got EOF from 127.0.0.1:57700 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 03:21:01.967424 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.960095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.14.227.189:33849)
0924 03:21:01.960298 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.960303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.960769 (+ 466us) server_negotiation.cc:407] Connection header received
0924 03:21:01.961029 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.961035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.961168 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.961240 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:01.961881 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.963140 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.965194 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.965904 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.966024 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.966600 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:01.966745 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:01.966836 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.967019 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.967102 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 03:21:01.976924 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:01.974784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.14.227.188:33845)
0924 03:21:01.974965 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.974971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.975484 (+ 513us) server_negotiation.cc:407] Connection header received
0924 03:21:01.975658 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.975664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.975765 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.975872 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:01.976702 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57716: BlockingRecv error: recv got EOF from 127.0.0.1:57716 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:21:01.990552 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:01.983740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.14.227.189:33849)
0924 03:21:01.983919 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:01.983925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:01.984394 (+ 469us) server_negotiation.cc:407] Connection header received
0924 03:21:01.984590 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:01.984596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:01.984685 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:01.984787 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:01.985211 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.986430 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.988412 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:01.989098 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:01.989229 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:01.989685 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:01.989852 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:01.989981 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:01.990126 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 03:21:01.990205 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 03:21:02.002104 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:01.999966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.14.227.188:33845)
0924 03:21:02.000156 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.000162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.000674 (+ 512us) server_negotiation.cc:407] Connection header received
0924 03:21:02.000816 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.000821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.000918 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.001025 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:02.001840 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57722: BlockingRecv error: recv got EOF from 127.0.0.1:57722 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 03:21:02.022914 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.015802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.14.227.189:33849)
0924 03:21:02.016022 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.016029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.016710 (+ 681us) server_negotiation.cc:407] Connection header received
0924 03:21:02.016904 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.016909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.017016 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.017119 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.017588 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.018782 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.020749 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.021484 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.021601 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.022167 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.022297 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.022409 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.022544 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.022621 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260924 03:21:02.032328 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:02.029900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.14.227.188:33845)
0924 03:21:02.030184 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.030190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.030714 (+ 524us) server_negotiation.cc:407] Connection header received
0924 03:21:02.030888 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.030895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.031020 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.031119 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:02.032092 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57734: BlockingRecv error: recv got EOF from 127.0.0.1:57734 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
I20260924 03:21:02.051460 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.044467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.14.227.189:33849)
0924 03:21:02.044649 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.044656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.045159 (+ 503us) server_negotiation.cc:407] Connection header received
0924 03:21:02.045411 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.045417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.045505 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.045633 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.046085 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.047252 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.049220 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.049917 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.050061 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.050612 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.050760 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.050859 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.051033 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.051118 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 03:21:02.059746 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:02.057539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.14.227.188:33845)
0924 03:21:02.057701 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.057706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.058261 (+ 555us) server_negotiation.cc:407] Connection header received
0924 03:21:02.058439 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.058445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.058537 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.058641 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:02.059495 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57742: BlockingRecv error: recv got EOF from 127.0.0.1:57742 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:21:02.087508 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.080023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.14.227.189:33849)
0924 03:21:02.080249 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.080258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.080840 (+ 582us) server_negotiation.cc:407] Connection header received
0924 03:21:02.081062 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.081070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.081168 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.081284 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.081759 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.083119 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.085174 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.085922 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.086080 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.086685 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.086839 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.086936 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.087114 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.087207 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260924 03:21:02.102139 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:02.099779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.14.227.188:33845)
0924 03:21:02.099964 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.099971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.100521 (+ 550us) server_negotiation.cc:407] Connection header received
0924 03:21:02.100705 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.100710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.100816 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.100936 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:02.101816 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57758: BlockingRecv error: recv got EOF from 127.0.0.1:57758 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 03:21:02.147552 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.140151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.14.227.189:33849)
0924 03:21:02.140389 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.140400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.140976 (+ 576us) server_negotiation.cc:407] Connection header received
0924 03:21:02.141216 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.141223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.141323 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.141444 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.141997 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.143190 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.145298 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.146038 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.146179 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.146667 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.146805 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.146901 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.147081 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.147174 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 03:21:02.157452 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:02.155004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.14.227.188:33845)
0924 03:21:02.155199 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.155208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.155760 (+ 552us) server_negotiation.cc:407] Connection header received
0924 03:21:02.155953 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.155959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.156080 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.156198 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:02.157182 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57766: BlockingRecv error: recv got EOF from 127.0.0.1:57766 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 03:21:02.230988 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.224815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.14.227.189:33849)
0924 03:21:02.224970 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.224974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.225828 (+ 854us) server_negotiation.cc:407] Connection header received
0924 03:21:02.226092 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.226100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.226189 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.226291 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.226726 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.227483 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.229326 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.229718 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.229802 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.230330 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.230431 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.230495 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.230642 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.230699 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 03:21:02.238760 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:02.236888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.14.227.188:33845)
0924 03:21:02.237040 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.237044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.237394 (+ 350us) server_negotiation.cc:407] Connection header received
0924 03:21:02.237569 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.237573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.237643 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.237765 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:02.238542 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57778: BlockingRecv error: recv got EOF from 127.0.0.1:57778 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:21:02.375260 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.370161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.14.227.189:33849)
0924 03:21:02.370413 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.370420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.370727 (+ 307us) server_negotiation.cc:407] Connection header received
0924 03:21:02.370972 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.370976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.371045 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.371145 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.371569 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.372260 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.373579 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.374037 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.374162 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.374572 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.374683 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.374779 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.374933 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.375010 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 03:21:02.378849 5798 catalog_manager.cc:1259] Loaded cluster ID: c2cd013b50ab4077851445016d842a19
I20260924 03:21:02.378976 5798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: loading cluster ID for follower catalog manager: success
I20260924 03:21:02.382625 5798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: acquiring CA information for follower catalog manager: success
W20260924 03:21:02.383112 5784 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:21:02.381069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.14.227.188:33845)
0924 03:21:02.381272 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.381280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.381555 (+ 275us) server_negotiation.cc:407] Connection header received
0924 03:21:02.381709 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.381713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.381778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.381865 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:21:02.382845 (+ 980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57790: BlockingRecv error: recv got EOF from 127.0.0.1:57790 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 03:21:02.384743 5798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8628b971804d4f9cbdb8db9b74cfcec2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:21:02.649498 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.643625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.14.227.189:33849)
0924 03:21:02.643857 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.643865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.644010 (+ 145us) server_negotiation.cc:407] Connection header received
0924 03:21:02.644224 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.644229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.644311 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.644391 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.644855 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.645634 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.647770 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.648206 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.648304 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.648717 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.648841 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.648914 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.649087 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.649186 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 03:21:02.660616 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.655306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.14.227.188:33845)
0924 03:21:02.655467 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.655472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.655763 (+ 291us) server_negotiation.cc:407] Connection header received
0924 03:21:02.655896 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.655899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.656050 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.656141 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.656442 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.657171 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.658657 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.659052 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.659158 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.659596 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.660129 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.660247 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.660335 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.660408 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:21:02.673518 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.666089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.14.227.190:40775)
0924 03:21:02.666627 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.666636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.667290 (+ 654us) server_negotiation.cc:407] Connection header received
0924 03:21:02.667528 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.667536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.667761 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.667955 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.668618 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.669484 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.671329 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.671748 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.671889 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.672378 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.672825 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.672945 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.673081 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.673169 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":374,"threads_started":1}
I20260924 03:21:02.674726 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.666116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.14.227.188:33845)
0924 03:21:02.666440 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.666449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.668078 (+ 1629us) server_negotiation.cc:407] Connection header received
0924 03:21:02.668347 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.668353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.668438 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.668525 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.669315 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.670189 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.672550 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.673197 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.673308 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.673899 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.674094 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.674185 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.674401 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.674475 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260924 03:21:02.675274 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:21:02.667433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.14.227.189:33849)
0924 03:21:02.667597 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:21:02.667600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:21:02.668000 (+ 400us) server_negotiation.cc:407] Connection header received
0924 03:21:02.668282 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:21:02.668289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:21:02.668374 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:21:02.668457 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:21:02.669288 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.670388 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.672611 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:21:02.673144 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:21:02.673256 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:21:02.674043 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:21:02.674174 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:21:02.674272 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:21:02.674951 (+ 679us) server_negotiation.cc:299] Negotiation successful
0924 03:21:02.675020 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 03:21:02.678274 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 5566
I20260924 03:21:02.679658 5628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:02.792284 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 5566
W20260924 03:21:02.813201 5731 connection.cc:535] server connection from 127.0.0.1:57670 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:21:02.813776 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 5640
I20260924 03:21:02.814970 5702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:02.922253 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 5640
I20260924 03:21:02.942005 15246 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1C3avF/build/asan/bin/kudu pid 5714
I20260924 03:21:02.943153 5776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 03:21:03.048992 15246 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1C3avF/build/asan/bin/kudu with pid 5714
2026-09-24T03:21:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4562 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8944 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (355688 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST